alt Hacker NewsQwen3.8 27B scores 52 on Artificial Analysis
Comments
And once again, Qwen 3.8 27B beats Opus 4.6, what the hell.
It's both funny and a bit terrifying and I still can't quite believe it. It runs decently on a gaming PC! Opus 4.6 came out only 6 months ago and was then broadly considered the new SOTA by a comfortable margin! How in hell did they package capability in the ballpark of a Feb 2026 frontier SOTA into 27B?!
More importantly, what's the point of building monster-scale data centers on unprecedented amounts of debt when a more than good enough model runs on a GPU from a couple years ago?
The coming months are going to be exciting, that's for sure...
I used this a lot over the weekend, and it's a really intelligent and strange model.
It gets really agentic at the higher reasoning levels. It does the basics like goal tracking and tool calling well, but more than that, it gets obsessed with solving problems and will do insane/unusual things to get to the solution. It actually reminds me of GPT-5.6-Sol-max which is similarly obsessive.
It doesn't surprise me at all that it outscores Opus 4.6. Opus had way better world knowledge but was more "human" with agent stuff - sort of lazy and uncreative, basically giving up once the obvious solutions failed. These newer models work more like magic, they are so creative and persistent at finding ways to get to the solution.
I used Qwen 3.6 27B extensively (>1B tokens) and DeepSeek V4 Flash (the older one also 2B+ tokens).
And I just can't fathom that the new 3.8 beats the new DeepSeek V4 Flash (which, in my eyes, is one of the best everyday coding models).
What an insane release, and convenient size to use every day/locally.
but i will test this model extensivly.
I have an internal automated benchmark, which roughly follows my workflow, and I've been testing various models on it, local and cloud. Qwen 3.8 27B did awesome. Its understanding is correct, research is better than e.g. glm's (and I like glm), and implementation is good and careful.
Qwen 3.8 27B doesn't look benchmaxxed. These "52 AA score" numbers feel real, which is surprising. I've been using it locally for a few days for other tasks as well. If not the speed, I'd be totally happy to use it as a daily driver instead of cloud models, it is that good.
--- (benchmark, to get an idea):
1. First, initial prompt which is not super precise - similar to how I'd write a task when talking e.g. to Opus. I'm describing an idea, and asking model to come up with some plan, and also to criticize the approach. Task is about implementing a particular pi extension. I'm checking if a model actually understands what I'm asking.
2. Then, as a follow-up, I ask to research alternative implementations, research UX of similar extensions, etc. It needs to do web searches, inspect open source codebases, read articles and papers, etc. I don't prompt to do this exactly, but I expect good models to figure out they need to do it.
3. Then, implementation.
Also, one finding: Q4 and Q8 seem to have very different behavior in this benchmark. Q4 produces 2-3x thinking in the end, and makes more turns - it seems it makes more mistakes, and needs effort to recover from them, while Q8 gets more things right in a first try. In the end, quality is roughly similar, but Q8 gets there much faster, especially the implementation (tried it several times). Could be a difference between concrete artifacts, or between runtimes, I don't know, but be careful - it seems the real-world experience with qwen 3.8 27B can be vastly different, depending on how it's set up.
Regarding DeepSeek 0731 vs Qwen 3.8 27B. On this benchmark, Qwen understand my intent better, it's better at research, and I also liked its implementation more. But: if you're more precise in what you ask, 0731 is also very good, and it's quite a lot faster on mac; raw speed is better, and it needs less thinking to get there. So, I'd say it's a tie in practice, both are awesome :)
The smaller these frontier-nearing models get, the more I'm reminded of https://en.wikipedia.org/wiki/Lottery_ticket_hypothesis
For more context, this puts it on par with models like GLM 5.2 and GPT 5.6 Luna, which are far larger
Why is it so small, but expensive?
Open Router
Input /M $0.45
Output /M $3.20
Cache read /M $0.05
Throughput 27 tps
It would be a very nice model at 200-300 tps and if it was dirt cheap. What's the limiting factor of optimizing speed and price for inference providers?
Under the same computing power, the improvement in LLM intelligence and the improvement in the upper limit of LLM intelligence are equally astonishing. At least in coding, the best local models that can run smoothly on a DGX Spark are now less than one year behind the strongest SOTA models in capability (I measured around 60 tok/s).
It's 7th (!!!) overall on the agentic index, above Terra.
China cleaned house these past few months. Kudos to them.
I would really like to see some open source US companies out there.
Why are hosting providers charging to much to host it, compared to much larger models? https://openrouter.ai/compare/qwen/qwen3.8-27b/deepseek/deep...
Perhaps model size and reasoning length trade off to some extent, similar to CPU vs. RAM. A smaller model with a longer reasoning trace has more intermediate structure to latch onto and build on.
I want to highlight its (1-hallucation rate) at 70%. BRAVO! For me, this is its most wonderful score. GPT-5.6-Sol sits at 8%.
I can't shake this the existential feeling that this compact series of 27G bytes represent something profound and universal.
I have a bad feeling about this.
US companies have spent hundreds of billions on their models, and they are not much better than the cheaper open Chinese models.
Perhaps they can out-compete them. If not there will be increasing calls to limit access to open models on the grounds of "safety".
Basically, if you can't beat 'em, ban 'em.
What's the best price/performance hardware to run it locally today?
Why is it not included in the Pareto line intelligence/cost chart?
The leveraged US labs are cooked. Debt's coming home. There will be bailouts.
I've been waiting on this model to show up on the Deep SWE benchmark results and treat its absence/delay as an indication of how slow and unusable it is for good results. I bet it thinks to the moon on some of those complex challenges.
The number I'd actually want next to every score: run-to-run variance.
A lot of the benchmarks seem often near meaningless these days - really bench-maxxed to the hilt. I tend to still look at the Artificial Analysis rankings to get at least an idea on relative performance of models, is that still warranted?
What or other opinions on how representative the AA rankings are of real-world performance? Any better indicators?
I'm impressed with the score. This is a model that runs on a good, but still regular, desktop PC.
Interesting to see where local models are going to be in the coming days. I am already starting to believe open source models are the way to go in the coming days. With Qwen 3.8 Max, Kimi K3 etx already delivering at part perf with frontier models, the future is going to be exciting.
Anyone try the 9B/2B distills yet? Wondering how they do for local tools
Very interesting. I was not expecting anything close to this.
Imagine this, and sucesor models on Cerebras or other silicon...
Unbelievable. Bravo Qwen team.
So it's effectively on-par with GLM 5.2 and GPT 5.6 Luna?
It tied with Luna/max. Simply incredible.
A model that I can run on a single consumer GPU is as intelligent as gpt-5.6-luna? That's insane.
[flagged]
[flagged]
[flagged]
Monster pure platinum cable for $100,000 per yard scored 3.1415927 on the audiophile benchmark!
As a comparison, Qwen3.6 27B scores 38, which was the highest in its small model category (4B–40B).
Qwen3.8 27B beats all medium models (40B–150B). It has the same score as DeepSeek V4 Flash 0731, which ranks #5 in large model category (> 150B).
Sources:
- https://artificialanalysis.ai/models/open-source/small
- https://artificialanalysis.ai/models/open-source/medium
- https://artificialanalysis.ai/models/open-source/large