It's not just about that, though. Have you looked at many women's bodies? Some are almost like men: broad shoulders, narrow hips. Some are like boys: narrow everything. There are "hourglass" shapes, "pear" shapes, big boobs... so many different shapes.
It might be possible to have a one-dimensional sizing scale for men's clothing, but it seems impossible for women's. There's at least 3 or 4 dimensions that are independent and nothing to do with being fat.
It's true, everyone can be not fat. But I don't blame people for this. I blame fast food and us not cracking down on food addiction as a society.