I believe Dr's have a place. With that said they often judge on the side of caution.
Maybe at your next visit or before you go ask him to put you in a waterproof cast if he feels that you truly shouldn't ride with it protected. This is kinda the best of both worlds. It may limit your riding to greens and blues but better than not riding.
Just a thought