logoalt Hacker News

seizethecheesetoday at 5:28 PM1 replyview on HN

The same thing holds for speed. I could build a system that speeds up Fable on GPQA Diamond ~50%, while improving score, by literally randomly selecting between Fable and Gemini 3.7 Flash. (Solve time for Flash is 0.1min and 0.8min for Fable, with Flash having a better score.)

Hell, I could publish better score at 87.5% time reduction by having the router always pick Flash!


Replies

alsimatoday at 6:54 PM

Like I mentioned, we omitted routing from the benchmark and evaluated with our strongest model modality. Also, randomly selecting between Fable and Gemini 3.7 Flash wouldn't preserve quality.

show 1 reply