Writing Rust code that's fast by asking agents to make the code faster

I have done a fair amount of low level performance optimization with Opus 5 and its reasoning is still very poor. Like why is CRC so slow and going through loops until I ask it if is using hardware instructions and it tells me it is using its own hand coded implementation poor. Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.

I've been having great results with this kind of thing. I find that really just need a sensible framework within which the optimization can take place. Essentially just providing the measurement harness, and some sort of motivation for what I'm doing.

The great thing about LLM is that it seems to have the checklist for everything. If I rattle off a few things like "don't allocate on the hot path" and "remember to pin the cores" it will come up with a few items of its own that I might have forgotten.

Eventually, it will have gone through the whole list with me, while having documented all the measurements along the way.

But it's still guided by experience. If I see unusual numbers, I might say "hey did you forget to compile it in release mode?" and it will apologize and fix that. If I don't, it may just continue exploring without realising everything is wrong.

If it can be measured, then LLMs can optimize it.

Once I had repo commands that could dump `sample` results and a cpu profiler/trace and then a benchmark tool that let me A/A + ABBA/BAAB-test the current modified git workspace against HEAD or any commit, the LLMs could just do their thing.

And that's how my homemade terminal uses much less memory than ghostty/kitty/iterm yet has more throughput.

AI is going to increasingly unmask people and companies who don't care about correct and performant software now that it's become so trivial to guarantee both. It used to at least be expensive and time-consuming and expertise-demanding to do those things.

can LLMs write better code if you keep asking them to “write better code”?

One of the keys for me was the use of types. Typestate when functions mint witnesses that can only come from it and are required to proceed and newtypes where you use custom types instead of strings so the agent can't forget. You can also use it to force the agent to use the implementation rather than reinvent the wheel by simulating linear types. Types are a much smaller target to optimize and provide constraints that fail loudly at compile time.

I have found that having a separate agent (session / instance) do the benchmarking and reporting the results back to looping optimizing agent is a clean way to prevent cheating. The benchmarking agent has no reason to cheat, its goal is to just to run benchmarks when tickled.

I also found this is really nice for quality evals. Have one agent with no context on how something is made do a quality review, with lots of detailed feedback. Then pass back the review notes to the implementor for feedback. It works a lot better than having an agent self-evaluate its own quality.

I've found that this is a good way to introduce bespoke code into the codebase whose perf gains don't generalize.

Unless its an easy memory/parallel/algorithmic win, its not worth it.

I'm finding the bottleneck writing Rust code to be that compiling and running tests is too slow compared to TypeScript - which is unfortunate, because I want all of the guarantees of Rust - I just don't particularly care if it runs at the fastest speed possible - at least not at the prototype stage, or ever for a lot of what I'm doing (though I do want the guarantees regardless).

I wish it could run interpreted to massively speed up development, and that it had a Mull-like mutant testing framework so that mutants didn't take hours and tens-to-hundreds of gigabytes of space.

They're quite good at just iterating different "ideas" on a performance metric with an objective measure. They can use tools like `perf` and do some analysis on the output. Sometimes they go off in the weeds unproductively, and sometimes they give up because your goal was too high, but as long as you're sort of babysitting the process, you can make pretty rapid improvement to naive code.

I keep the benchmark harness checked in and read-only to the agent, and let it touch everything else. Once it can't rewrite the measuring stick, the gaming just stops.

AI is heaven for bullshitters, bloggers and (fake) benchmark addicts.

As the author of Tera mentioned in the article, I am curious how it can get 2x faster. Was the benchmark using tera v2 with the `fast` feature enabled?

Tangent: are "LLM" and "Agent" synonymous?

Lot's of talk about how LLMs are bad at optimizing. And they are. They are still fundamentally semi-learned stochastic parrots. But LLMs are related to the Infinite Monkey Theorem - we've given monkeys typewriters, but pretrained the monkeys to be better than random.

They don't have to be good if they're cheap, directed, and sufficiently random to hit on something good within your budget.

Are LLMs better than our top mathematicians? No way! But they're able to spitball and falsify in parallel way faster. Eventually their directionally random output will hit something novel and interesting.

[flagged]

one thing nobody mentioned here, once the agent is looping against the same benchmark it will happily optimize for the benchmark itself and not the real workload. worth rerunning the win against a slightly different input shape after, just to check it did not memorize the harness instead of actually fixing anything.