Thats not exactly 'validated'. Feels very noisy, it is not a good bar for either - does this code do what the user actually asked - is this code actually 'good'

There would be so many examples of coding projects that these models began or attempted to work in, that were abandoned because the models were floundering.

I would imagine the labs have some decent ways to produce novel requirements and then actually validate they are met, without the noisiness of implicit human feedback.

That said, the more I think about it, you are right, there's probably also very good ways to extract signal for all these sessions.