hey im in the same boat.. im a short guy also only 5'4" and find pants extremely big. If you can find pants in xs they usually fit fine. I say as long as the waist fits the length isnt too big of a problem since the leg gaiters usually dont go below your boots anyway. Many companies make pants with some kind of leg cinch or some way of shortening the length of the pants when walking around.
Personally Id rather have a slightly loose fit, which isnt a problem since most pants small pants can fit someone whose 5'8". When your riding having the extra length isnt a problem since it isnt going anywhere when you strap on. Layering is much more comfy if the pants are loose. You may think it will look a bit silly with all the baggyness of the pants but when you put all your gear together its not so bad.