alt Hacker NewsAs is standard for academic papers, this is an attempt to describe the state of the world as of when they did the research. But then it takes time to write it up and get it out, after which newspapers report it as current events.
In this case, we're looking at a picture from 2025. The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5. And the paper's design means you're mostly looking at the very initial effects of deployment, before companies developed good review processes.
Replies
They've updated it for the agentic code era, and have found basically the same thing. Code can be created faster but that has moved the bottleneck to review.
By that logic, it is not ever possible to do research on LLM-assisted work.
> The models involved in the main agent results were primarily Claude 3.5 through Sonnet 4.5, GPT-4.1 through GPT-5, and Gemini 2.5.
What we have today is not meaningfully better than what we had a year ago, so that isn't relevant.
The study was updated twice with the last time on August 2026.