Sounds like you are on the east coast too.
Yea, you really have to watch other people and be careful when passing. Sometimes you just have to either ride the mountain early or late to be able to really bomb and go fast safely.
It's always better to check your speed and slow down and give a wider margin. Whoever is ahead of you always has the right of way, even if they are going slow and wide.

on wearing a helmet, saw a kid break his in half today.