This is an incredibly silly comparison. It amounts to claiming that a Ford Pinto is just as good of a car as a Rolls-Royce by simply observing that both cars got a person from point A to point B. After all, once someone reaches their destination you can hardly tell what vehicle they actually used to get there, but that doesn't mean there's no difference between vehicles.

What matters most in state of the art models isn't simply the final destination, it's the process of how one arrives to that destination.

Correct output is table stakes. Your test only shows that the products work as advertised, it doesn’t reveal reasons why people prefer one to the other.

You’re guessing that it’s a result of advertising, and I agree that that’s probably a component, but it’s a mistake to assume that they are interchangeable when you have people saying to you directly “I use both and they’re not.”

If I were to give one carpenter a set of fine hand tools, another a full workshop with power tools, and they both made a picnic table to the same spec, and at the end I wasn't able to tell which came from which, would you say I have come to a fair metric for which type of tooling to use for wood working?

To add on context, the experiment you're giving is called a *blind judging test*. Remove the branding and labels, and let judges sample the results and see if they can tell which is ranked correctly.

Some examples are blind wine tasting tests. There are instances whereby some journalists invited renowned/established wine tasters and subjected them to blind wine tasting tests. Turns out the judges couldn't tell which was which. Pretty embarrassing.

It speaks volumes as to how people can accurately judge the value of things. There is research by some network scientist that says you can't generally can't tell the 1% from the top, though you can tell the really bad from the generally good. What OP's experiment might tell us is that the LLM competitive advantage is so small no one can tell which is objectively better.