alt Hacker NewsI go off of marketing obviously, but how is every new announcement of any vendor always on top of all others and then cheaper, while next day the next best and even cheaper model pops up? Is there anyone who independently verifies this? Its kind of pointless to comment on this announcement, much like how its pointless to comment on all the other announcements...
Replies
Because they selectively choose which benchmarks to highlight and which competing models to compare against; they don't always compare against the SOTA or the Pareto frontier of performance/price, a lot of times you'll see them compare against the last generation of competing models, or the like, so that they look better if you don't carefully consider what they're comparing against.
For example, here you see they are comparing against Opus 5 and GPT-5.6 Sol, when Opus 5.5, Sonnet 5.5, GPT-6 Astra, and GPT-6.1 Sol would be the latest SOTA models from those labs to compare against.
Also because a lab's own benchmarks tend to run about 6.4% better than third party runs of the same benchmark. Source: https://whatmodel.sh/claims This can happen due to a number of factors, and doesn't necessarily indicate anything nefarious. It could just be that they run multiple times while fixing issues that aren't related to the model, and have a selection bias towards stopping when the score is on the higher side. But since you see that sometimes their own scores are lower than the independent runs, some of the variation is explained just by variance in the model solving problems, since LLMs are nondeterministic there can be variation from run to run and you don't see labs always cheating, just potentially a bit of selection bias about when they stop tweaking their benchmark setup and publish the results.
And yes, there are independent third party runs of benchmarks for just this reason. The most popular with broad coverage of a range of benchmarks is https://artificialanalysis.ai/ but there are a number of others as well.
You have to read between the lines (or wait for someone on HN to do it for you).
It's never "cheaper and faster and more intelligent", it's pick 2. Sometimes (as was the case this month), it was "cheaper per token, but reasons five times as much", so your costs go up.
I run a benchmark and independently check all new models. They are very spiky.
If you are doing coding only then official benchmarks are mostly right, but as many people are starting to do for instance marketing work with LLMs now there’s very little correlation between coding and creativity for instance.
This seems worse across the board compared to latest DeepSeek Flash, except on some automation bench.
So, I can quickly judge that this is an interesting option only if cheaper or faster than DeepSeek.
Based on Terminal Bench 4.0, I'd also guesstimate behind GPT 5.6 Sol, so of course behind the SOTA as expected.
I have this idea that the big labs internally have much more capable models which are prohibitively expensive to run though, and the competition is mainly around getting the price down by releasing models that are quantized/otherwise made more efficient at the cost of a lot of performance.
I've got a corollary question, which is "how big is the market for these never ending model announcements?" ... I guess its mostly people coding with agents. Most general AI users are just asking ChatGpt to word an email for them or whatever. How many AI users are on here looking for the next model announcement and working out if the price per token is better? Are we talking like 50,000 developers? More? Less?
It's always cheaper (*than the next more expensive model) and better (*than the next worse model).
And often they also don't mention how verbose it is, that the same text splits into more tokens than in competing models, how slow it is, that the pricing is temporary, that the cost per task actually isn't that good despite being both cheap and capable.
There are hundreds of benchmarks (all noisy), dozens of model families, and dozens of modes and variants within each. Announcements are crafted by publicity teams and involve cherry picking a chart that looks compelling.
As a prospective user, none of of the benchmarks details will matter to you, but the family of models and settings they peg themselves against can give you a hint of where you might want to evaluate them within your own workflows.
As an enthusiast or lay researcher, the more interesting thing is to look at their techniques and any innovations they're exploring in training, inference, etc. Those are usually going to be details tucked below the fold and won't be summarized in a benchmark chart.
As anybody else, why do you even care about this in the first place?
This doesn't just apply to AI. You should know better than to take marketing announcements at face value, nor trust that most "influencers" or "journalists" or "commenter" repeating and discussing those announcements are doing a lot of work as they buzz over what's said.
Don't be a victim to PR spin, but also don't discount potentially interesting work just because it got channeled through a marketing team. Just look for the information you need amidst the noise or move on if there's none there.