> Even worse, it's not a fair comparison: they purposefully just used "adaptive" instead of "max" for Fable.

We agree models should be compared on a fair basis. Unfortunately, adaptive was the only publicly available number. Anthropic doesn't generally let us run their models for evals, so we rely on whatever Anthropic or third parties have published. In this case, the Agents' Last Exam leaderboard has Fable Adaptive, but not Fable Max.

https://agents-last-exam.org/leaderboard

Would have loved to publish a full curve for Fable if anyone makes the data available.

Although we do bias toward publishing evals where we're ahead, we have historically been unafraid to publish evals where we're behind (e.g., GDPval). The point is give people useful information to decide what's best, not to trick people.

Edit: Now I see there's a second entry with xhigh effort. Not sure if that was added or recently or we skipped it.

(I work at OpenAI.)