alt Hacker NewsInteresting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment.
Worse, they don't control for breaking the application's tests.
> Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on.
Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.
Replies
At best the conclusions are a best case scenario for degraded code quality. That things remain functionally OK with more costly token usage to get work done.
> they don't control for breaking the application's tests.
Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.
That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.