Just layer with clothes that you can easily take off with the exception of your baselayer. I find that my UA ColdGear compression mock and insulated jacket do the job 90% of the time (mostly board in 20-30 degree weather). I'll do the occasional light cotton long sleeve over my coldgear. I do wear a T-shirt most of the time just because I don't want to walk around the lounge in only my UA.
By the way, thanks to everyone who recommended the UA hood in the other thread. I ordered 5 of them for my girl and her fam. Got some socks from them too