Does code cleanliness affect coding agents? A controlled minimal-pair study

Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment.

Worse, they don't control for breaking the application's tests.

> Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests already present in the repository, and a cleaner-side and messier-side solution that both pass the hidden test may still differ on tests they were not graded on.

Any conclusions with respect to token consumption seems pretty meaningless if we're not controlling for the quality of the final output.

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs just getting it right at the 1st/2nd attempt.

One trick I've found that works well is to tell it to refactor, e.g for Python:

    Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions.

    The Rust codebase in `/src` has become bloated with several files >1k LoC. Refactor the Rust codebase to fit code organization standards expected of popular open-source Rust code without causing any benchmark performance regressions.

In terms of benchmark performance it generally improves after the refactor which I suspect is coincidental (especially in Rust where it shouldn't matter due to compiling) but I'm not complaining.

In similar work we're conducting at NJIT, we've found similar results. (We call it "Contextual Quality Contagion") Some of the most interesting comments here are about some of the real-world scenarios that are commonplace in industry: "mixed" quality codebases, as well as codebases with a mix of legacy code patterns with newer "better" patterns, causing the agents to confuse the conventions.

The minimal-pair design is honestly one of the strengths, as it tries to isolate the cleanliness from other factors (e.g. architecture, dependencies, tests) which is more rigorous than comparing repos. However, using LLM-generated "sloppified" code (rather than some kind of mechanical or human-guided) is a bit questionable.

I'd say the biggest critique which others have correctly mentioned here is the authors' choice to not check the full test suite. The claim of "behavioral equivalence" is only as good as the tests and coverage.

This hypothesis is compelling for two reasons: 1) it makes sense (garbage in -> garbage out as it were) since LLMs will mimic what they see in the codebase; 2) it matches what many engineers feel somewhat intuitively over the last year or two of using these models. Greenfield is almost always easier than joining a busy codebase. The mess comes in complex integration, and maintaining a system for legacy purposes, etc.

"agent pipelines that [...] clean a messy [repository]"

This feels like a terrible approach, sufficient to condemn the entire study.

Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.

I can't imagine how it wouldn't. None of them can fit a real codebase in context and have to browse the code the same way a person would. Doing searches and reading files. If the files are in the places they would be expected to be and things are called what the model or a person would first guess to search then it gets found in the first attempt rather than requiring a deep search and multiple attempts.

Even if agents can learn to navigate all the stubs and WET crap they leave behind do we really want a code base that no human can follow what's actually going on?

Interesting to see this quantified. Clean structure seems to lower the cognitive load for both humans and agents, which probably explains why naming and modularization matter more than we think.

A lot of those tokens are traversal - either searching for code or following call-sites. Basically building enough context to be able to work on the task.

You can reduce a lot of the token use for traversal by giving your agent access to some form of LSP in addition to hierarchical direction with your AGENTS.md (or equivalent) for monorepos - but a spread-out codebase is always going to end up requiring some form of traversal to solve each task.

And that traversal isn't just token use - its repeated round trip latency (LLM (queue time -> prefill -> decode -> output) -> Agent (parsing -> tool call -> tool response) -> back to LLM) for EACH step (well, some can be done in parallel, but in practice its mostly sequential) - slowing down the task considerably.

Locality and structure are key when it comes to efficient use of agents. The context window is always bounded and attention across it is inconsistent.

Did the degraded variants also end up with weaker test coverage, or was that controlled separately? Because the fix for "agent burns more tokens understanding the code" is very different from "agent gets confused by dead code" — the first needs better tests and docs, the second just needs a linter pass.

In the real world, clean code still has value.

In AI labs, they can run a task repeatedly for extended periods, eventually marking it as completed in benchmarks. But in the real world, completing tasks quickly and well remains important and valuable.

In my experience everything that affects engineers affects an agent. Good abstractions, reasonably sized methods, good names, principled (intra & inter) service architecture, unit tests, etc.

All of these things have historically been the job of engineers, because it helps other people contribute to the code.

Now it helps other people and other agents contribute to the code.

I wrote this, where possible I use non llm tooling to assess and llm just writes up the findings.

https://github.com/bjcoombs/ai-native-toolkit/tree/main#what...

If the models are quantifiably different on cleaner codebases, surely this could be leveraged into a measure of code cleanliness.

I'm not sure if you would call it an objective or subjective measure, a fixed model would be consistent which would provide an objective base for comparison, but other models would be different, so it would be subjective in relation to the model itself.

I would assume- yes, if your tokens transport inexperience a inexperienced answer is reflected back to you? Of course self assured as if absolutly right.

Another consideration: written by hand, the trade-off of development velocity vs well organised code means that it can be worth taking some tech debt now in order to deliver some value now. (Especially when prototyping etc.).

With coding agents, agents can produce code quicker. The same trade-off still applies.. but, the time it takes an LLM coding agent to write well organised code is still going to be quicker than the time it takes me to write scrappy code.

Agents struggle with this: DRY means they create helper functions and classes for a testing scenario and maybe it's called two times. When you ask for a refactor of your code and tests to follow, these helper functions are also ignored and then, dead code starts piling up. Specific cleanup sessions always seem to leave residues behind and doing Ralph loops in the first place seems to help with this, but I'm just not satisfied with the overall performance. Any ideas?

As pure slop coder, after telling agent to clean up my messy projects from earlier models, i.e. ts, 400 line max, and most of my tasks have gone down to consuming fraction of quota as previous and tasks takes significantly less time to finish. I also just tell it to evaluate tasks on compute/token efficiency, i.e. whether to reuse context, or smart new convo/switch models, and now my quota stretch much further. Like I assume agents should be smart enough to route tasks to proper model capabilities in background, but then I ask myself... what economic incentive to they have but to make users token max.

imo yes, based on my experience with 4+ repos in production built using AI.

1. clean structure means the AI can predictably work because i want deterministic output.

2. basic things like following SRP leads naturally to layered code which makes testing new features and fixing bugs super simple.

3. reviewing code is simpler, because i see ai agents as "servants" i command to do something and thus an overseer is still required

the study found clean code helps agents. my takeaway: the agent is better at reading my code than I am, and that is genuinely humiliating.

agents are pretty good at cleaning up a codebase, finding dead code, fixing bad abstractions, etc. You just have to spend some focusing the agents on that goal.

Research done by SonarSource, a company that sells static analysis products. This explains the focus on static analysis, among other things.

If we don't write the codebase, we don't write the tests, and the agents are modifying it, what even considered a messy codebase? Too many files? Too little files? Related code spread accros many files? What is considered a messy codebase?

Have you even seen one ever?

Also, not checking if it breaks unrelated tests is wild, good software is written by modifying the "just right" amount of code to get your result.

If you break _unrelated_ tests than you most likely didn't changed the right amount of code.

Idk I call bs.

[flagged]

[flagged]

[flagged]

[dead]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

"Across 660 trials with Claude Code, code cleanliness does not change the agent's pass rate. However, it substantially alters the agent's operational footprint: agents working on cleaner code use 7 to 8% fewer tokens and reduce file revisitations by 34%. Our findings suggest that traditional maintainability principles remain highly relevant in the era of AI-driven development [...]"

[flagged]

[dead]

[dead]