I get a lot more than that.
24VAC will rectify to 32VDC. You now have 20V to drop in a linear regulator at 80mA which is 1.6W. A smallish heat sink should be enough for that, though. So if it is getting very hot, I agree with your conclusion that something else is wrong.
Bob