alt Hacker NewsWhile this is very impressive benchmark-wise, GPT-6 Astra showed us that benchmarks don't always correlate 1:1 to intelligence of a model.
When Astra launched, I think Artifical Analysis showed that it was on par with GPT-5.6 Sol and lower than Opus or something like that? Then, they updated the scoring.
I hope that more open source models, including this model, to be "as good to use" as Astra.
Replies
walrus01 • today at 8:37 AM
Apparently the scoring on a lot of difficult benchmarks can also be extremely influenced by something as simple as waiting for the model to exhaust its reasoning, realize it hasn't come to a conclusion yet, and give it a simple prompt like "you can do this, I know you're capable, please keep going".
I don't know why, but the benchmarks still fails to cover the difference between large models and small ones. The small ones are great for many things, including general coding, but the larger ones, like fable and astra, have some kind of intelligence that is not present in the small ones.