You need something that is (a) resistive, not a simple semiconductor, so it doesn't make a huge distorted mess of the audio signal, and (b) smoothly and continuously variable over a fairly wide range of resistance, so the attenuation can be smoothly controlled.
Using an LED and a LDR, the LDR...