alt Hacker NewsWhy tho? The whole point of agentic is that it takes several passes to get to a result. If step a) is a non-idiomatic transpile, but step b) c) and ... z) make it idiomatic and performant and efficient, why look at step a)? That was just the beginning. The idea with "agents" and "swarms" and dynamic this and that is that you only look at the last step.
This strictly follows the early agentic usage, where people were complaining of code full of errors and such. But having the loop do plan-write-verify is the thing that got us from that state to today. And it works. If you look at the code and it has errors (linting, etc) then your loop is not properly setup. You should only be called upon when all that can and should pass does. Your role is "direction/taste/human-common-sense-at-a-glance".
Because the point of the test is to see what the agentic swarm approach is capable of. If your test assumes that a version of the thing you are trying to build is available in the training data, or can't determine if it matters whether the thing you're building is in the training data, then it's not a very useful test. Most of the time when you consider agentic approaches, you're not trying to rebuild something the agents involved are trained on.