I have to say I don't envy this process but it can be done. I'm handy, have journeyman electrical and can build just about anything out of wood or metal but a house comes with so many hookups, code and tricks of the trade I just wont take it on... I'm having a house buit for me right now, the only thing I'm doing is the fence (if I even do that). Realistically I just don't have the time to kill myself over it, working evenings and weekends for extended amounts of time after full days on the job browsing forums... no thanks lol. Thats me though...
That being said I know three people in the last two years that built this way.
1. went pretty smooth and on budget, they had connections to trades and a lot of help. Was about 2 months late and caused them a lot of stress but they managed.
2. Came in a year late and 100K over budget, almost destroyed their marriage and the house is not up to par, they would never think about doing it again.
3. Her brother ^ - Started a year after her, was done in 6 months, has a bigger house and came in 150K under them... is completely happy with it.
Prepare yourself for the worst if you don't have relationships with trades you might run into a lot of scheduling issues and I guess you need to weigh your ability to deal with stress and no free time... for me I have a low tolerance for both.