The 3 towns mentioned above are going to be your best bet for a middle sized town.
The problem if you want to be away from the crowds, obviously, you will have to find a job in a smaller town. You have some options like Steamboat, Aspen and Durango which are all prohibitively far from Denver for most of the crowds and as such have less (but way smaller towns).
If you DO decide to make Summit county your spot for riding, you will have many more options of where to live/work. Denver has alot more in common with Sacramento then it does with LA fwiw, and there are alot of nearby towns and suburbs with their own vibe. Evergreen is a great spot, but similar options are conifer, genessee, boulder, golden, idaho springs, georgetown. these 'burbs so to speak are all on the western side of town putting you close to your jet up i70 but within easy commuting of anywhere in Denver.
I realize you said not Denver, but Colorado is alot like Nebraska and Iowa with mountains. There ain't shit outside of the main town. Now of course this is entirely untrue, but you have to appreciate the tradeoff. Its always going to be easier to find housing and work in a metropolitan area like Denver. Summit county works but during the season is more hectic then Denver is anyway.
People like snowolf don't like CO at all, and I love it, so take it fwiw. A little info on my geographical biases: grew up in Manila, Philippines (20+ million) with a home base in SoCal and relatives in Amador county (near Sacramento). Went to college in Santa Barbara, then lived for 6 years in Chicago and 6 years in Omaha. Now been in Littleton (southwest Denver right against the hills) for 5 years and have no intention of ever leaving (other than perhaps moving up the mtns a little to one of the towns I mentioned). Denver is a chill town and CO is a pretty chill state. Mid season can get pretty hectic but shit, summit county probably sees more visitors/season than anywhere, there are just tons of acessible mountains.
Dont be put off by summit county crowds either. Places like Copper and Loveland never get the crowds that Vail and Breck get, are loved by locals and highly underrated. I purposely work weekends and ride weekdays, also a consideration.
TLDR: Durango would probably be your best bet, but much more challenging logistically. Dont be scared of Denver it has everything for everyone, lots of nearby towns with their own vibe. You guys are gonna love it here.