I went a couple weekends ago when it was pretty chilly in CO. It wasn't bad.
Layer up!
I typically stay home if it's really shitty though, there's always tomorrow! The coldest place I've rode was in Montana during our trip. It had be 30 below zero and pretty windy too. We were the only dumbasses on the mountain.
Was a fun day though once we finished off the snapps and went numb.