Assuming a phone battery capacity of ~3.3Ah at a voltage of 3.7V, full charge requires >12Wh of energy. If you can sustain, say, 3W output from a hand-cranked generator it will take over 4 hours to charge the phone, so it's not surprising it's a struggle. The buck-boost converter is a good idea...