What if you are really short for your weight. I think height does matter a little bit. To maintain the proper stance on my board, I have to adjust my bindings way in the middle of my board and that was barely. According to the "guidelines" I should have a 19-20" stance but I can't get it closer than 21" or so on the inner most holes.
The reason people say height doesn't matter is that there were times when rental rats would just grab a rental board for someone and as long as it was around the chin or nose they said that was a fit. Still happens today, but not nearly as much as it used to. That style of "fitting" is basically just to get people through the lines as fast as possible instead of getting the best board for your riding type and best size of that board possible.
Height does actually matter IF you are very tall or very short, but that is about it.