I don't understand why that idea was adopted that IA had to be that "give one command have a perfect run and correct answer" thing when it's clearly not how it works and an eternal "the last % is worth 99%" issue.
I know why the US frontier labs push it of course, you can't push a trillion dollar plus idea if you also say it should be a cheap commodity and the best and latest are now the answer, a bazillion mid levels are.
But most work of value I do is with a swarm of "not the latest nor the brightest" models, with a smart one controlling the dance. Most decisions or review is better done through consensus. The myth that "if we keep going it will reach a point where it's always correct" seems so obvious to me, it doesn't work like that, for that matter we don't work like that.
This is not a jab at it. I work with it. I love it. But I think the story imperative are a mismatch for the actual needs of most companies, and it's hiding between the "frontier needs" of special research & co, or the governemental needs.
On the same line of thought: it's pretty obvious as great as these things are, no they're not going to "replace all employee", and no employee without them won't be able to compete either. They're a supercharger, a self multiplying smart assistant with capabilities in any field, and it feels so wrong to me that they've been presented in a complelty different way.
I recently tried some personal evals to see how some different models performed. I was trying Opus, Sonnet, Gemini 3.8 (this was pre-Argon) and a couple of others. I picked a pull request I'd recently made with heavy Claude assistance, one that seemed simple enough to be quick to solve but with enough subtleties that I could differentiate the results. Set up the agents to run clean without knowledge of the solution, and set them loose.
To my surprise, none of them came close to producing an acceptable PR. Quality varied, but even the very best results were missing significant pieces.
My takeaway is that the human in the loop is still massively important for many things. And that makes it really hard to come up with a useful benchmark.
[dead]
> it doesn't work like that, for that matter we don't work like that.
The parallels with self-driving cars are interesting. We demand that agents don’t just produce better code than most humans (which the frontier models do), but that they perform much much much better.