Yes......it's not the idea that will not work but your mechanical setup.
If the hall was mounted at the end of a brass tube and the magnet was mounted the end of a bolt for inertia the magnet would always approach the hall in the correct alignment.
If you want the magnet to arrive higgeldy...