Hello,
I think that a LM7805 will not drop the voltage as wanted.
It needs even 1 Volt difference at 0 mA and 1.5 Volts at 20 mA:
Better use a LD1117-50, that will have a voltage drop of 1 volt at 100 mA.
Also put the motor direct on the battery.
This will save you from a voltage drop when...