Well that's two misconceptions that need to be dismissed. The first one,... that all audio signals are AC is untrue and the second, that both conductors need to be switched is also untrue.
Regarding the first misconception; yes it is true but only the signal that is driving the speakers. The...