Yes and no. It would make the test fair but it would also mean losing any optimizations the vendors have made specifically for their model. They're all trained differently so they should be used differently to get maximum benefit. It makes sense to use each vendor's harness to get the most of their models, and compare the results that way. Additionally, it's not easy to use all the models in all the harnesses. Anthropic's subscription only covers claude code for example.

> I’d suspect the harness to massively affect token use and optimisation

Yes they do according to databricks -> https://www.databricks.com/blog/benchmarking-coding-agents-d...

That's why I find comparing models on benchmarks only gives the tendency. We should be comparing model x harness to have accurate metrics.