I have a couple. A smaller one that I used to use at school and an Osprey Kode 38 that I was gifted.
Typically carry a GoPro (don't always want it on my helmet), tool, clif bars, jerky, gatorade, extra goggles/gloves (stuff happens, it's helped before), and that's about it. On a typical day. My friends don't like to stop so we just ride and ride and ride, eat/drink our snacks on the lift and ride again.
The Osprey I'll use on those days where I'm hiking a bit more. It's nice - a bit gnarly for me but it was a gift and I'll be damned if I don't get the proper use out of it.
It might be a bit cumbersome at times (it took me a few runs to get used to it) but on a full day of riding all over the place, I'd rather have it than not. I don't always know where I'll be riding / hiking / traversing with my friends, so I'd rather put up with a little discomfort and have what I need to keep myself fueled for the day.
As far as whether it affects my riding, I haven't noticed it.