alt Hacker NewsAt that point aren't you just edge-case testing?
Surely most of your use-cases are not novel tasks that combine obscure domains.
It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.
At that point aren't you just edge-case testing?
Surely most of your use-cases are not novel tasks that combine obscure domains.
It seems to me the real way to evaluate the value of a model is how it performs in your real-life workflows.
Certainly, real-life is the ultimate benchmark. But for various reasons that isn't always immediately possible to go evaluate a model on.
Maybe my Greek idea sounded too high falutin' or simply seemingly clever (I give an example below -- try it out!).
So here's something way simpler that Qwen3.8 27B does not get; only GLM-5.2 and K3 do.
**
Analyze the 2 structural (not semantic) patterns in this text:
1. Morning revient.
2. Birds saluent Morgenlicht.
3. We suivons Waldwege toward maison.
4. Rain tombe plötzlich; we cherchons Schutz beneath sapins.
5. Night vient langsam; we trouvons Wärme near le Feuer, sharing quelques Geschichten together.
**
The answer should get not just the obvious cyclic E-F-G pattern but also the word counts being Fibonacci. Surprisingly few models get this. The only way I got Qwen to do this was on the 2.4T model, with extensive prompt scaffolding. Claude (Opus & Fable), Sol, Grok etc. got it on the first attempt. (All models, all attempts max reasoning level.)