Hey Kellygoose, sorry I was offline.
The point I was making is the 5v rail has more amperage. Hence the slower motor speed. As all have said here already, the drill motor probably draws a considerable amount of current to even start turning..
The above suggestion has some merit too.
Some power...