By all means mine isnt a wedding ring or anything so not as fussed if I lost it, but I'm always taking my gloves off to do things like change music etc and never once thought it was going to come off. Most guys rings should be fine.
You're supposed to take your rings off during any physical activity, reason being in case you break your fingers. If your finger gets twisted/swollen you will have an issue. I always take mine off and leave it in the room/home.
I have fat knuckles which tend to expand and contract (rest of my finger doesnt change much). So the correct size for me means rings slip off if my hands get very cold and get tight if very hot. I personally take mine off as the snow will mean it is loose and could potentially fall off.