Does the B7 A4 audi not have pin points for the rack? I can't remember if my B7 had it but my B8 has points for the thule rack to "grab" into and I think it definitely helps keep it secure.
My guess is the distance between the towers was too wide, I had that happen the first time I put mine on and it definitely felt loose at a stop. At 65 I can see how if one side moved a little it would throw the whole thing off...
I think this is universal style rack where you buy the correct "mount feet" and they latch onto the car as opposed to the oem votex racks that lock into the pegs like you described.