If you're wearing all that and are still getting cold, then maybe a down (instead of fleece) midlayer would help? Down jackets still keep me way warmer than any synthetic or fleece I've tried. The only problem is that the nylon fabric might not let sweat pass through as well as fleece, but you said you don't sweat much so it might not be a problem for you.
Also, if your jacket has an insulated hood that's big enough to go over a ski/snowboard helmet then it can keep your head warm on the lift, especially if it's windy. I bought a Flylow jacket this year with a very large hood and it's been a game changer.
Do you start out the day warm? How about bringing a bottle of hot tea in the car so you can have a few swigs of hot beverage in your insides before you step outside?