"short" is not unfriendly or rude per se. That said, I'm not sure you would call a person "short" to his/her face, which is weird to me, because no one has ever hesitated to call me "tall" to my face - which I am, and since about the age of 13.
Maybe it's not a good thing to be short? .... in terms of society. I vaguely remember there were studies and stuff at one point about how very successful people, well, men (in business in particular), were often tall. Societal prejudice?
If I were addressing someone directly, I would likely use (very) petite for a woman, or "you're not that tall" for a man. (euphemisms much? lol)
As far as "vertically challenged" goes - definitely not freak-show level, but agree that it was maybe a couple of decades ago - it was always meant tongue-in-cheek, likely in response to the "this challenged" and "that challenged" considered politically correct at that time. I would only consider it humorous, never insulting. That said, you would only say it directly to a very close friend who totally "gets" you. :)