alt Hacker NewsThe point of ARC is essentially an "IQ Test" for AI systems. It is meant to cover abstract reasoning capabilities of generally-intelligent systems like LLMs. What the author did here was build a system that only solves ARC problems.
The other tension is the fact that this score is on the public eval set. In machine learning, you typically have 3 datasets: training, evaluation, and test. The training set is the dataset that's used to update the weights according to your loss function, you are "encoding" the patterns from the training set directly into your model. The eval set is what you use to track performance while training, it is NOT used to update model weights, but shows how well the model generalizes. The test set is a private holdout set that is only used when you're "done" developing your model. The difference between test and eval is information leakage: you can use performance against the eval set to modify your hyperparameters and model architecture to get better eval scores. So while the eval set doesn't directly update the weights, it can indirectly cause "overfitting" by tailoring your model to do well on the eval set. What you really want to see is the private test set performance, not the eval set. For all we know, this model could be ridiculously overfit on the eval set and perform poorly on the private test set.
Replies
I am very new to this but applying human intuition this still feels like cheating. Knowing all the question that will be on the exam and working on understanding them even if you are never given answers will obviously give you and edge.
Does ARC measure "one shot learning"? I heard that the major unsolved problem in ML was developing systems that are good at dealing with novel problems.
It seems like an interesting strategy. Based on the author’s comment, they haven’t been at it for very long. So, I guess the folks who run the private test haven’t had a chance to get to it? It’d be interesting to hear how it does.
Instantly one of my favorite comments this year. Nicely done.
They have _not_ trained on the test set.
On the private test set, the right way to evaluate this type of model, is giving i it the test question Q, which it will first train to AR predict first, and then it will inference using the just-updated weights with Q as prompt, giving you back A, and then you compare A with A_true secretly.
2 different things are being called leakage here
1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, instead of being frozen after training
2) overfitting: agree that true measure is private set. It scores on private set roughly on par with TRM (a comparable model), obv with much lesser compute
--
also re LLMs: they do not follow this 3 split since (a) Incredibly hard to keep a pretrain dataset clean, (b) common in labs to benchmaxx during postraining (and known to do so on ARC)