I think the Hero3 battery is slightly smaller, but not that much smaller. Is your Hero3 set to 60fps? That would probably be a drain. Wifi is the other battery drain.
I basically only turn wifi on for aiming my camera and then turn it off. I'm also using mine in temps 35* and below which kills the batteries faster. It's surprising to me because its an action sports camera and that includes a lot of snowsports. You'd think they'd consider that when thinking about battery management. It'll be interesting if I can get more recording time mountain biking in the summer time.
I still really like mine so I'm willing to deal with it. Of course, last time out I put a huge fingerprint on the lens while changing batteries on a chairlift. didn't take any video after that.