Heh this story made me laugh.
I wouldn't rule out wearing L size, it will depend largely on the manufacturer but I weigh 180 and can wear a pair of 36 pants without a belt, normal clothing I buy is pretty much awlays L but my snowboard pants are XL and I would not want them to be any smaller at all. They are O&E brand tho, not Burton.
Pants that are a little baggy are fine as long as they stay on your hips, a pair with decent adjustment straps that you can adjust without any folding/snow inlet issues will pretty much fit regardless.
Pants that are too small suck tho, Ive got a feeling going down to a medium will be too small for you while L will be slightly big (ie: ideal)