How much speed are you looking for? Kawasaki has the new Ninja 300 coming out that looks very promising as a commuter and can be had for under $5k. That said I would place the Ninja 650 and SV650 basically equal. As said, see what feels best. Though I would take the SV simply because I dislike the way parallel twins sound. My current/first bike is a 1983 Honda Nighthawk 650. I'm probably getting an SV650/Gladius next spring. I keep trying to convince myself to buy the Ninja 300 but I bought a sensible car (2012 Mazda 3 skyactiv) I can't have too sensible of a bike.
Remember that the smaller displacement bikes are lighter and can be tossed around like a hooker on her prom night. Nothing wrong with the sub 600 series bikes as fun toys! In fact because you can wring their necks without breaking the sound barrier, I'd say they're possibly MORE fun than bigger bikes.
I think the perfect size would be a 400 cc inline 4. Something like the old FZR400: They had 60 HP (close to the SV650 and Ninja 650), but weighed about 350 pounds instead of 400+, and to get at that power you had to wind them up. Would make for a fun bike!
Damn you all are making me want to get back into the sport, and an FZR400 is what I want.