As davenn hinted at, the problem is the amount of power that the attenuator will have to dissipate. The extra energy is dissipated as heat. You need to tell us the kind of power levels you're working with.
Even wirewound potentiometers have limited heat dissipation, unless you can find a custom-made large wirewound potentiometer - I expect they would exist somewhere.
A feasible solution would be a switched resistor network, using wirewound resistors (with heatsinking if necessary) and a rotary switch (or several). If you want to go that way, you need to specify your impedances, power specifications, adjustment range and adjustment resolution.
A simpler solution would be to use a separate small amplifier for your remote speaker and control the volume at its input.