Actually, I think you're really underestimating how many people struggle to find clothes that fit them well. Most people's bodies are not perfectly average. For clothes not to fit well, you don't need to be a significant outlier in one dimension. Just the accumulation of several smaller deviations from average can be enough to create an awkward fit.

Beyond that, even if we limit it to height alone, there are hundreds of millions of people who are much shorter or much taller than average.