Never without. My current gloves have this shim at the top (Burton) and for me it never really gets in the way. Of course I can't bend my wrists back as I could without... But thats the friggen point.
The argument of "It limits my mobility" is bullshit. Same goes for back protectors. There is no movement you can't do with one on.