It comes with time. You're afraid of going fast right now because you don't have good enough control and a high enough comfort level on a board to go fast.
That's a good thing. The self preservation instinct is a strong one. Keep riding and you'll get there. Until then, be safe and ride within your limits - for your sake and everyone else's. Travis Rice didn't get on a board and start throwing double corks right away ya know.