Factory wax is basically just a storage wax. It's just thrown on there to keep the base from drying out during shipping and while the board is sitting around the shop.
grafta, are you sure you're getting enough wax in the base? For the first wax, I always go super heavy on the wax. Sure, it leads to more waste, but for that initial wax I really try to load it up.
You're not gonna hurt your board riding it on the factory wax.