The simplest way I could think of, would be to make a small circuit that puts out 6 pulses for each button press. You'll need 2 of them, one for each button, up and down.
When you push the button, you'll advance the counter in the potmeter 6 counts instead of 1, and pressing 5 or 6 times max...