alt Hacker NewsThe problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more.
Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.
He used LOC, and it isn't bad. Just this week I had a LLM do a small task, and it produced 500 LOC, every little detail beautifully abstracted out. But 500 LOC for such a simple change looked suspicious to me. As everything must pass human review, I re-wrote it to see what happened. The result was 100 LOC. No human wants to review 5 times a much code.
The issue really isn't "there is no good metric" - when I saw 100 LOC vs 500 LOC for the same thing there was no argument. The problem is Goodhart's Law. Whenever we use a metric use like LOC as a reward function for humans, the result is a disaster. I have no doubt that's true for LLM's too.