It comes with time. You're afraid of going fast right now because you don't have good enough control and a high enough comfort level on a board to go fast.
That's a good thing. The self preservation instinct is a strong one. Keep riding and you'll get there. Until then, be safe and ride within your limits - for your sake and everyone else's. Travis Rice didn't get on a board and start throwing double corks right away ya know.
"People say that marijuana smoking is going to get in the way of my career. I say to them that on the contrary, my fighting career is getting in the way of my marijuana smoking." -Nick Diaz