So after trying to stay squared up, I can definitely tell that was the major problem. I never realized it but when i would start to drift my shoulders were opened more to the front of the board and my hips were following. After letting it drift for a second I would pull my shoulders back to be in line and my tail would swing back into place. That being said I can say that I've spent way too long riding open like that cause it was weird as hell riding square.
Thanks for all the help guys!