alt Hacker News2 different things are being called leakage here
1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training
2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute
--
also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)
> weight update during eval: this is a form of test time training and not really cheating.
Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.