alt Hacker News> weight update during eval: this is a form of test time training and not really cheating.
Possibly "not really cheating", but it does make benchmark comparisons unfair - especially as the other models are unlikely to have their weights updated during the eval.
no, most models on kaggle are finetuning during test time, (including LLM based approaches)
Pure frontier LLMs dont, but thats because nobody knows how to make it work cleanly and at scale. Once someone makes it work, it will be deployed