I would say it depends. If there is a reasonable chance that the average person on the mountain would mistake you for a ski patrol guy, then you really should not be wearing that outfit. If it is clear that you are not ski patrol, then it is a douche move to make you take off/change the jacket.
It is a judgment call and I do not think we have enough information to make that judgment (not having seen the jacket, etc.).