logoalt Hacker News

fancyfredbot • yesterday at 4:09 PM • 0 replies • view on HN

You are Google, training a new model. You regularly benchmark the checkpoints. It's gradually improving as you train. When do you stop training and release the model?

You certainly don't do it when you are just behind the frontier. Instead you wait until you are ahead on a lot of benchmarks.

So, if Google release a new frontier model, the chances it's ahead on most benchmarks are statistically about 100% because if it isn't yet then they won't release it.