> Perhaps you are reading too many benchmarks.

Maybe, but at least the benchmark provides an objective measurement of the codebase it is tested on. You're also assuming the bugs are newly introduced / regressions.

I can give a concrete example - Fable will not interact with bugs that result in writing to null pointers in C code. That triggers the guardrails and ends the session. If Luna (or GLM Flash, etc) will fix those kinds of memory bugs, that immediately puts it ahead of Fable in some ways, no matter how tiny Luna is. Again, models are spiky.

I still agree with your initial point! It's LLMs all the way down over here. It would need to be a particularly gnarly bug & an exceptionally talented human for me to want to pay another human to work on fixing it now.