Ok I am thinking something along these lines.
The only concern I have is about the power adaptor, we could put another diode in line with it's power lead depending on the type you choose. But if the battery is flat and you plug in the power adaptor and you try to use the amplifier then it could...