Depends on how you ride. If you do large turns on a well-groomed slope you're fine. If you're getting into rough terrain where you need to turn on a dime and make lots of tiny adjustments to your line you might have a problem.
You're basically getting a split-second delay between what you want your board to do and what it's actually doing. If you find your edges bite more than they should that might be why.
Have you tried adding a second pair of socks?