First question to ask is what is the voltage range allowable for your "digital" devices.
For your circuit to work it should be something like 2.25-5.25V ,is that so?
Second question is ,Do you need the entire "digital " circuit working when the USB isn't connected or just keep the RTC running...