Ante,
Your experiment inspired me to try it too, and then I continued experimenting:
1) New 9V alcaline battery feeding a 7805 with no load. Current drain = 6mA. Its been going a day-and-a-half now, and its battery measures 7.8V. The regulator feels cold and its output measures 5.05V.
2) I made another one but put a 51 ohm load on it. That's 98mA into the load and the regulator was using only 6mA. The regulator was dissipating 406mW and felt slightly warm. At 4 hours, the battery measured 6.6V, and the regulator output still measured 5.05V.
3) I built a zener circuit with a 1N4733, 5.1V/1W zener, and calculated a series resistor to feed it and feed the load. With a 51 ohm 100mA load, I figured that with a 6.6V battery then a 15 ohm resistor would feed the load 5.1V. With a new 9V alcaline battery, the 15 ohm feed resistor got VERY hot since it was dissipating 1W. The Zener also got hot and was dissipating 816mW. At first, the total drain on the battery was a whopping 260mA. The battery voltage dropped to 6.6V in about 2 hours, when the zener felt cold, and the load voltage measured 4.4V. Lousy regulation.
4) With a new 9V alcaline battery, I tried the zener circuit without a load. Again the drain on the battery measured 260mA, but the zener was smoking a bit since it was dissipating more than 1.3W. The battery voltage measured 6.6V in about 3 hours, when the zener felt warm, and the output voltage measured 5.0V.
So with a new battery and 98-100mA load, the regulator dissipated 406mW, and the zener/feed resistor dissipated 1.8W.
And with a new battery and no load, the regulator dissipated 24mW and the zener/feed resistor dissipated 2.3W.
The zener regulation can be improved by feeding it even more current.
The zener/feed resistor costs more than a 7805 and eats power, so why use a zener? Maybe it's better for low current applications.