alt Hacker NewsCan someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
Can someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
This is not actually running the Arc-AGI-3 anymore. To summarize TFA:
1. The AI plays the game and records outputs.
2. The AI does TDD using those outputs to create its own copy of the game.
3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human.
4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time.
To make an apt comparison... You go to get your chess ELO. You don't know chess at all and you're really bad at it, so you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.