Fast =/= Out of control. Most of those guys can avoid traffic better than you think.
Having gotten run over by expert skiers a few times (fortunately only board contact), I'm not so sure I agree. Each time, there was ton of space to pass me.
There are a lot of morons who think they are cool if they zap by you leaving maybe 1ft of room. Not that they are completely out of control, but carving at 50+mph, they aren't going to make sufficiently quick course corrections.