Actually, I might have found a better solution. 1N5401 diode seems to have a forward voltage drop of 1V, so I basically end up with 5V after the Diode, and since it's a diode, it can stop the batteries from being connected backwards anyway as a bonus :)