Biggest problem I've noticed with a binding that's too small is problems strapping in. The straps have to be on the biggest detent and there's very little extra strap to guide into the buckle. No biggie when you're sitting in the shop, but when you're on the slopes with gloves on and snow under your boot, it's a bit of a struggle.
I'd go back for the L's.
Coffee is your friend. Everyone else is an acquaintance.