alt Hacker NewsStill, basic debuging and trouble shooting is where LLM generally shine. Any model can bisect git history and isolate newly introduced bug.
If model can not automatically reproduce bug, while human manually can... you got a problem in CI.
> Luna Max performing better than Fable 5.1 High
Perhaps you are reading too many benchmarks.
Edit for answer :
I agree Luna is great cheap model.
But if Fable was hitting security limits, yet was still included in benchmarks...
What flies better? Elephant or paper plane. You can make objective benchmark about that. But not much value for logistics company
> Perhaps you are reading too many benchmarks.
Maybe, but at least the benchmark provides an objective measurement of the codebase it is tested on. You're also assuming the bugs are newly introduced / regressions.
I can give a concrete example - Fable will not interact with bugs that result in writing to null pointers in C code. That triggers the guardrails and ends the session. If Luna (or GLM Flash, etc) will fix those kinds of memory bugs, that immediately puts it ahead of Fable in some ways, no matter how tiny Luna is. Again, models are spiky.
I still agree with your initial point! It's LLMs all the way down over here. It would need to be a particularly gnarly bug & an exceptionally talented human for me to want to pay another human to work on fixing it now.