Wow. I hope no such "audiophile" falls in your hands
I couldn't agree more, I mean who cares about the fifth harmonic which will be better filtered by the bypass capacitors anyway. One of the extremes I have seen (can't recall where, prob some advert) is a silver/rubber ring to put around valves that "reduces noise and distortion by allowing electrons to travel smoothly from cathode to anode". ???
Regarding batteries, they will have lower noise than any regulator, no matter how good it is. However, as you said, their high internal resistance makes them a bad choice for hi-fi audio.