If you are using fets, the gates need to be pulled positive and negative. An emitter follower will only pull positive so a resistor, gate to ground, should be fitted to pull down, try 10k.
I attach a picture of a suggestion for a simpler inverter and for two versions of the output switch, both...