> I imagine for shorter men, it's the inverse but equally bad.
Not really, as a quite short guy, many shops will offer me to have the clothes fitted, and if not it's pretty trivial to fit them myself. Maybe on the most extreme end of short it's more of an issue, but in general I suspect shortening pants and shirts is signficantly easier than lengthening them.
I'm a 5'5", 110lbs man. I shop at the teens section and get larges. I may not get the trendiest looks, but I get cheaper clothes that fits and looks good on me!
I also tried Stitch Fix, they had a surprising amount of stuff that could fit me (both fashionably and size wise), albeit not as cheap as kids' clothes.