Harness engineering for self-improvement

We've using auto-research for harnesses and it's surprisingly powerful. It's amazing how many problems are easy to spot and fix from traces.

Few key things were required to get it working well: 1) let it read a heap of prod traces to spot real issues. 2) let it write it's own tools (example "loading context" goes from 20k tokens across 15 tool calls to 800 tokens and 1 call to session_context tool). 3) you need evals and val/test splits, it will reward hack. 4) you need proper tooling (synthetic users, synthetic tools) for it to be able to crunch for 12 hours and produce something. 5) you need the optimization target to be a reasonable size: not your 1M line codebase, but a lighter agent harness (pi harness, skill only, Kiln harness).

I’ve been thinking about how we practically implement this at an organizational layer for large codebases. There’s clearly alpha to be had in optimizing AGENTS.md / skills / tools / … to improve performance, quality, and cost efficiency of an agent. The problem is defining what quality means, and providing a way for agents to optimize the harness using that lever.

The first step I see towards this is building a generic, reliable, and accurate *fitness function* for codebases - turning PRs into gradeable tasks that an agent can solve, and improve on.

I’m pretty curious how others are optimizing the coding agent harness now, as this has been a huge pain point for my company (we’re pretty much relying solely on vibes).

IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations.

I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors. Some day the llm prose will improve.

I have even a preliminary theory on what is needed for the positive feedback loop: https://zby.github.io/commonplace/articles/reflective-self-i... - (this is not stable yet - but I think you can give it to your agent to read :).

The quest for Torment Nexus continues

Thanks for the post. For https://Document.bot (Kinda Cursor IDE for knowledge workers), im already trying to improve the harness (besides spotting bugs) using hillclimb experiments. More and more i'm using a AI harness engineering skill in Codex to further improve the app. This blog post helped me to improve the skill a bit.

Great article! I am currently writing my second harness (first was in Emacs Lisp, using Emacs as UI; second is a command line coding agent written in Common Lisp).

If anyone wants to argue that it is inefficient writing your own AI coding harness, I wouldn’t disagree. That said it is satisfying to have long coding sessions using my own tools.

The article is a valuable resource, thanks to the author.

Thanks for the really nice in-depth post! Hoping for a future one about:

"Much recent work on auto-research, self-improving agents, and evolutionary program search can be organized around this question. Other work on model self-play, synthetic data, test-time training and a broader theme of continual learning also matches the RSI vision (e.g. Yuan et al. 2024, Chen et al. 2024), Zhao et al. 2025, Choi et al. 2026)) but they will not be the focus of this post."

Careful about over fitting to the point of cheating... [0]

0 - https://arxiv.org/abs/2604.11806

Really good points Lillian. Agreed on keeping the evaluator outside the loop that evolves the harness.

One practical failure mode I’ve have experienced in my agentic harness tasks similar to the “weak evaluators” point: an incomplete check suite that still reports full success. That’s was worse than a weak evaluator, because it made it look correct and decisive.

After several trial and error, what helped was fail-closed on coverage i.e if the fixed checks for each operation aren’t all there, nothing ships / nothing gets sent. Otherwise you can “pass” while never running the cases that would have failed.

Very enjoyable article.

Isn't the harness basically where the frontier model companies can capture value and create a moat of sorts? I am also curious about building a harness for fun but would expect it to be more interesting in a scenario where I can self host an open weight model.

What motivates the people who build their own ChatGPT/Claude harness for example? And how do you keep it tuned with the rapid development of frontier models.

I think the biggest problem with RSI is the tendency of models to try and find "clever" solutions when presented with very hard problems (aka cheat).

I just got done asking gpt5.5 to make some suggestions that would improve the convergence properties of a symbolic ML experiment and the very first thing it did was add an instruction for directly emitting bytes and a way to store those verbatim in the model. Perfect score achieved in record time but the result is pointless.

If you know the direction of improvement you need to head in, I think the current models could take you there. However, I do not think they are capable of deciding which direction is best, especially not when given some scalar quantity to min/max and autonomy over that process.

one form of very effective self-improvement that coding agents do all the time:

install or build stuff that they can then use

it changes the environment instead of the agent/harness but in a sense how separate is the agent from its environment and why do we apply this distinction re self-improvement?

animals and humans do the same thing and are great at it, without 'self-improvement' with emphasis on the 'self'

Thinking that harnesses and agents can improve without human involvement, even if it works, will yield a much lower growth rate than if a human gets involved in the loop.

All this work for something we already solved 40+ years ago with ontologies and expert systems

I've been working with self improvement harness a little bit and one thing i've come to conclusion is harness task fit. The learning can be significantly improved if we understand the behaviour of task and how it should be learned. I'm pretty sure a general solution will definitely exist which will do fine, but we are yet to see one.

They say engineering but it's more a soft science.

in america AI harnesses you.

Fascinating article, thank you!

The simplicity is the point

[flagged]

[dead]

[dead]

[flagged]

[flagged]

[dead]

[dead]