alt Hacker NewsYeah but what is the point of "naked model" evaluations? It seems to me that the only interesting question is capability per time and cost (and conditioned on other important things, like privacy). It doesn't matter to me which part of that equation is being implemented by model weights and which part by other supporting software.
Replies
timschmidt • yesterday at 7:07 PM
Harnesses can fairly easily be adapted to other models. Moving capabilities from one model to another is a more involved process requiring distillation, training, etc.
➕ show 1 reply
It is useful to compare like for like.
Currently if my hypothesis about frontier labs doing creative tricks between the model and the client is true (and the results seem to favour it so far) the benchmarks are giving us an artificially lowered results for open weights models.
I have yet to test opus/sonet via my proxy. If Qwen gets 10% better and Opus stays the same that suggests one if two things: - either opus doesn't need it - or it's already done behind the scenes.