Cold temps aren't the problem for me. It's the flipping wind. Gusts at that speed combined with those temps can suck real fast.
The wind at Stowe literally gave me the push I needed to ride my first black diamond many years ago. I was standing up top of the Front Four, planning on just riding by to do Lord or Ridge View, and a massive gust hit me just right and sailed me over the initial drop in, to the top of Hayride (or Haychute, depending on what map you're looking at). It was so windy I was more afraid to unstrap my board and try to walk back out than I was of the steep, so I said WTF and went for it. Loved it!