alt Hacker NewsI don't know about others, I can only speak for myself. But I do appreciate numbers for bare models, numbers for model + harness, numbers comparing different models in the same harness, and numbers comparing several models across several harnesses.
It's a lot of information to ingest, but it gives me some idea of which part of the system is doing which part of the work, how well different harnesses and models interoperate, and more insight into the part of the equation under my direct control as a software developer.
I agree that all else equal, the more comparisons the merrier. But IMO, there has been way too much focus on benchmarks for the bare models, when it seems to me that what actually matters is what you can do with the technology, and that is never limited to what a bare model can do.