Where I notice the speed difference between me and you guys is the speed that you guys are comfortable jibbing stuff or playing around. When I experiment with freestyle I don't feel comfortable doing so with much speed.
That comes with practice (obviously)
Generally speaking in freestyle speed is your friend because you don't need to balance as long, if you're going faster. But the flip-side of that is that any change in situation needs a much quicker reaction. Like, you notice you're off-balance... you need to do *something* and that something is to (as gracefully as possible) dismount, because you can't fight it, and if you don't dismount you'll eat shit.