I started when I was 23 and thought the same damn thing. We all do ya know, wonder why and regret we didn't pick it up sooner.
I just turned 27 and ride a board like I'm 17.
In my four seasons I've broken my neck/slammed and swollen up my back and just recently smashed my elbow into 30 broken pieces and half my arm is filled with metal. Never have I been broken so hard and never did I think I would love something enough to not care about the risks. The feeling of being creative, the adrenaline and feeling like you're living is what makes it so rad. It's worth every gain, pain or not. Hopefully you'll be less of a dare devil than I am. Good luck!! Happy shredding!!!