Seeing the dramatic differences in scores just going from high to xhigh is just another demonstration of the bitter lesson: Just keep scaling search and learning. We are probably going to need a lot more GPUs.

This is the first I have herd of this benchmark. Can someone explain how it in any way indicates how close we are to "AGI"?

Replay of Sol attempting the game: https://arcprize.org/replay/83543d22-8e1e-439a-8809-129ff1d9...

It seems a weird and arbitrary challenge for a language model to be expected to perform. It also seems like there are some harness/visual issues even in the first few steps, where it states that it hasn't moved when it clearly has.

We have it slightly ahead of Fable in our multi-agent coding evaluations.

Fable's main advantage is that its average solution size is smaller. However, GPT 5.6 Sol is a substantial improvement from GPT 5.4/5.5 which would write verbose, defensive code. 31KB for GPT 5.4/5.5 down to 26KB for GPT 5.6 Sol, with better performance for Sol.

Fable scores slightly lower, but with an average solution size of 12.2 KB.

Data at https://gertlabs.com/rankings?mode=oneshot_coding

> Cost/task: $25.1K

Yikes

I'm surprised it is that low. Are not all top AI labs "cheating" and workaround LLMs's low sample efficiency by hiring people to generate more data points - similar problems with answers, so they can train models on those and improve scores? A good benchmark for general intelligence probably should be a complete black box, no sample data given/leaked at all.

it seems the older models were capped at 10kusd for the runs though?

Very interesting. My prediction is that Mythos would outperform Sol.

Also what does this tell about Yann LeCuns whole world model theory? Bro has been going on and on about it. He has made multiple wrong predictions on the trajectory of LLMs.

At some point his claim should be fully falsified no?