Reading your quote really gets me wondering who the people making those kinds of analysis are...

It's like they haven't maintained any actual software, because the criteria they choose is... Completely irrelevant?

The things that matter are tooling, orchestration and ecosystem - as well as how the LLM will actually implement the solution for a task

LLMs constantly do idiotic things. If you have good libraries to utilize, the likelihood of the solution actually working goes up because they no longer need to implement the hard part.

If you have orchestration for dependency injection, code generation, meta analysis etc

Tooling like the way otel tracing is integrated, openapi generation etc is also invaluable because every time the LLM does something the likelihood of it being hallucinated/wrong increases etc

You'd need to implement a nontrivial project in different languages, then add nontrivial features across them and only then start by rating eg correctness and incident occurrence after the final output

But token use on a one shot? Completely irrelevant as far as I see it.