> Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.

I suspect a lot of the training set for this sort of thing is people online speculating about cache performance incorrectly.

Can you give it the valgrind suite to loop over? I have yet to try that with AI, but maybe the cachegrind tool is enough to help it.

I see the same thing, except I was working on high level performance reasoning. Whenever a piece of code has multiple steps that require multiple algorithms to work, AI almost always fails to guess which step is the slowest and what causes that step to be slow. Even Fable makes wrong guesses. You definitely need to give them a benchmark feedback loop.

Are there actually any humans who can reason about things like cache performance from first principles? I know there are some people who think they can, and I suspect they're fooling themselves. The one iron principle of micro-optimization at the level of cache hits is "measure, measure, measure", you just cannot think your way to the right answer on the first try. Processors and instruction sets today are too complicated, and tips that worked on one generation might be neutral or worse on the very next revision, making all the cargo cult knowledge passed around on this topic at best useless. I'll echo one of the sibling commenters here and say that LLMs probably bullshit their way to answers on questions like this because that's what humans online do as well.

If you give an LLM a proper testing harness and feedback loop to actually generate hypotheses, test them and revise them, I suspect it will do much better.