My first guess would be that you're trying to rotate your upper body to face downhill. Not a problem when you're on your heelside obviously, but as soon as you go toeside you'd be torquing your upper body around which will tend to turn your board downhill. Fight that for a few feet and I bet you faceplant.
Coffee is your friend. Everyone else is an acquaintance.