logoalt Hacker News

clueless • yesterday at 5:06 PM • 4 replies • view on HN

how much benchmark-maxxing has been done here? taking bets on poly and kalshi...


Replies

xynelius • yesterday at 5:15 PM

I think it's safe to assume every single model is benchmaxxed. I mean why wouldn't you include training data from common benchmarks? Not only is a higher score good marketing, but these benchmarks are supposed to represent common tasks so training for those should in principle improve performance across a wider range of tasks.

samseaborn • yesterday at 5:18 PM

There's an entire section in the linked post about measuring benchmaxxing, so I guess that probably says something.

➕ show 1 reply
phoghed • yesterday at 5:13 PM

All of them are doing all the benchmaxing they can most likely. Dumb not to when so many people care about those numbers and don’t have the opportunity to try every model out personally.

bigyabai • yesterday at 5:13 PM

Muse Spark was generally good. I'd be surprised if this model felt benchmaxxed.