There are two types of potentiometer 'laws' - linear and logarithmic.
For audio purposes (volume) it is common to use the logarithmic law versions - if you fit a linear version you find the problem you're experiencing i.e. 'everything happens at one end'.
In tone control circuits the opposite...