Damn, not what I wanted to hear... I'm definitely fatter than you as I'm out of shape 185lbs lol. Guess XL is my size... I wear size 34-36 jeans.
For me, it was more a function of my jacket being too tight around my hips than the shorts being too small.