> You're asking basically to ignore bugs and correctness.
Not ignore correctness, just bugs that will be caught by tooling.
> Can it be a useful comparison?
IME, yes. LLMs in an agent-loop are trivially able to write spaghetti code that will never do an off-by-one error or something else that is easily caught by tooling, which is not something humans can do.
Judging code on whether it has bugs easily caught by tooling is pointless - LLMs are running the tooling in a loop anyway, so no matter how bad or poor their code actually is, it never exhibits bugs that are caught by tooling.
alt Hacker News
> You're asking basically to ignore bugs and correctness.
Not ignore correctness, just bugs that will be caught by tooling.
> Can it be a useful comparison?
IME, yes. LLMs in an agent-loop are trivially able to write spaghetti code that will never do an off-by-one error or something else that is easily caught by tooling, which is not something humans can do.
Judging code on whether it has bugs easily caught by tooling is pointless - LLMs are running the tooling in a loop anyway, so no matter how bad or poor their code actually is, it never exhibits bugs that are caught by tooling.