After writing that I actually came across the tutorial that explains the logic behind it. You have to keep holding the direction, if you let go, next time you press it it will go in the opposite direction. This applies whenever you're on the sides or the back of the tower. It's crap, it's hard enough as it is.
Anyway, just beat floor 7, I have 75 lives so I think I'm doing ok, this is one game I wont be playing on hard though, I'll probably do the other playthroughs on very easy so I can get through the puzzles as quickly as possible.