logoalt Hacker News

wat10000 • today at 5:04 PM • 0 replies • view on HN

I recently tried some personal evals to see how some different models performed. I was trying Opus, Sonnet, Gemini 3.8 (this was pre-Argon) and a couple of others. I picked a pull request I'd recently made with heavy Claude assistance, one that seemed simple enough to be quick to solve but with enough subtleties that I could differentiate the results. Set up the agents to run clean without knowledge of the solution, and set them loose.

To my surprise, none of them came close to producing an acceptable PR. Quality varied, but even the very best results were missing significant pieces.

My takeaway is that the human in the loop is still massively important for many things. And that makes it really hard to come up with a useful benchmark.