As long as your leave through an appropriate gate or the ski area has an open boundary policy, they should not be taking your pass.
The uneducated guys you see walking from BC runs on the pass, most likely have no idea the situation they are putting themselves into. I know it's crazy, but even with all the awareness we have pushed around Berthoud Pass, I still see a ton of people without gear and not a clue as to what they are doing.
Typically if I get to chat with those groups, I give them my FOBP card and encourage them to check us out. Hopefully they will continue to either get on relatively safe backcountry terrain, or continue to get lucky. You can only lead a horse to water.
Also, keep in mind that most of the avalanche fatalities that have happened in the last few years, the victims have had education and gear. Of course they knew what the consequences were versus these other groups who may have no idea what they are getting into.