alt Hacker NewsI agree, however It's too easy to game arbitrary limits like "medium". There is no olympics for "just rolled out of bed and slightly hungover"
Replies
You don't need to directly compare the same reasoning levels. Today you can compare Sol on Low vs Haiku on Max, for example, but you can't look at GLM 5.3 High, only Max.
For some tasks I genuinely feel Opus 5.5 does better on low than medium or high. The trouble, as always, is figuring out which.
Edit: I realize after commenting that you're not suggesting medium is useless, but just arbitrary and differs between models. Personally I don't want to directly compare mediums between models, I just want to be able to compare across effort levels.
A graph of time (or cost) per task against success rate does it. Just include all the different levels.
That's the beauty of the line charts; the labels are arbitrary points on the line. You can easily compare the series visually.
We don't even need a standard for reasoning levels.
Just accept whatever the provider names as "medium" and compare the accuracy between models vs the time it takes to complete a task.