alt Hacker NewsI mean the y axis is deceptive to make it seem like greater gains since it starts at 30%, when in reality the differences aren't great.
Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.
What about the graph looked so unreal to you?
> Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.
We agree models should be compared on a fair basis. Unfortunately, adaptive was the only publicly available number. Anthropic doesn't generally let us run their models for evals, so we rely on whatever Anthropic or third parties have published. In this case, the Agents' Last Exam leaderboard has Fable Adaptive, but not Fable Max.
https://agents-last-exam.org/leaderboard
Would have loved to publish a full curve for Fable if anyone makes the data available.
Although we do bias toward publishing evals where we're ahead, we have historically been unafraid to publish evals where we're behind (e.g., GDPval). The point is give people useful information to decide what's best, not to trick people.
Edit: Now I see there's a second entry with xhigh effort. Not sure if that was added or recently or we skipped it.
(I work at OpenAI.)