alt Hacker Newsit looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it.
it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful.
what this harness does is get the model to write a simulator first, it's measuring something entirely different.
Replies
That is a fair assessment, but the "something else" is independently valuable, maybe even more so than model improvements - constructing an architecture for efficient rule determination and execution. In other words, I think the goal here isn't so much to beat Arc-AGI but to develop a generic improvement beyond "Ralph loop", which could dramatically extend frontier capabilities for all kinds of uses.
Writing a simulator requires understanding the game well enough to spec the simulator. Acquiring said understanding - within the action limits of the benchmark – seems like the heart of the challenge, so this doesn't strike me as "cheating" at all.
This is classic goalpost movement. Arc-AGI-3 was launched this year with roughly 0.5% success for frontier models. being able to 99% it in less than six months sets a new record for Arc-AGI saturation timeline. Speaking of singularity measures. It is definitely a big deal, not least in that Chollet needs to cancel his summer vacation and write Arc-AGI-4 now.
okay, what if this benchmark were just called Arc, and all you knew about it was what questions it asked? it just looks like a bunch of arcade games, which should tell you that it doesn't test AGI at all.
on the flip side, the idea that most tests are bad, even standardized tests, the tests that you scored well on that gave you all your opportunities in life: it cuts to the emotional, grounded core, the absolute foundation, of too many people. in the crowd of hacker news commenters; people who buy anthropic shares at retail; the people who work at tech companies; and their kids, families, etc., who are a bunch of nobodies, there are a lot more incentives to believe "stupid fucking arcade games test AGI" than not.
The simulator the model builds is comparable to the mental model of the game humans create. It is also much more efficient, GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3
> State grounding turns raw observations into objects, variables, and relations that can be tracked. Mechanism discovery finds how that state changes under an action and writes the rule as an executable program
The way I'm reading this isn't that they are writing a game simulator, but rather that they have two things they are evolving - a perceptual model of the game mapping from pixels to objects, and a behavioral model of how each action acts upon these perceptual objects. The behavioral model is written as a program that can be backtested by the game states and actions they have already taken to see if they are correctly predicting the resulting next game state.
The ARC AGI 3 games are non-trivial, and I think it's very impressive to see them doing well using this approach.
I'd agree with their conclusion:
> We read a saturated ARC‑3 as the new beginning: mechanism discovery as a general capability — grounding the causal structure of a world through the agentic loop of action and perception, in environments far richer than a 64×64 grid. This is where we are heading to.
This is the way that an animal learns about it's environment - by observation (and innate biases) to recognize the objects in the environment, and predict their behavior, both autonomous (which AGI ARC 3 doesn't test - the objects in the environment are passive), and in reaction to the animal's behavior. The animal predicts and observes, updating its predictions when it is wrong.
A system that could do this in a messy, dynamic, real-world environment would seem a like genuine step in the direction of animal intelligence, especially if it could ditch the symbolic representations.