My first guess would be that you're trying to rotate your upper body to face downhill. Not a problem when you're on your heelside obviously, but as soon as you go toeside you'd be torquing your upper body around which will tend to turn your board downhill. Fight that for a few feet and I bet you faceplant.
I told my kids, "Someday, you'll have kids of your own." One of them said, "So will you."
-- Rodney Dangerfield.