He used LOC, and it isn't bad. Just this week I had a LLM do a small task, and it produced 500 LOC, every little detail beautifully abstracted out. But 500 LOC for such a simple change looked suspicious to me. As everything must pass human review, I re-wrote it to see what happened. The result was 100 LOC. No human wants to review 5 times a much code.

The issue really isn't "there is no good metric" - when I saw 100 LOC vs 500 LOC for the same thing there was no argument. The problem is Goodhart's Law. Whenever we use a metric use like LOC as a reward function for humans, the result is a disaster. I have no doubt that's true for LLM's too.