alt Hacker NewsQwen 3.8 27B is excellent, but it defaults to overthinking things
Comments
All current era models overthink as it's a product of their RL incentives (or distillation of models with them...)
From my reading of the Fable 5 and Opus 5 System cards, my reconstruction is something like:
Finish the task → make externally observable evidence that it is finished → check your own work → fix problems → don't stop prematurely → satisfy the evaluator comprehensively.
That is fantastic for SWE benchmarks and autonomous agents. It also naturally creates pathologies:
under-answering is expensive; over-answering is cheap.
I had been planning to buy a GPU for Blender to compliment my 5950x CPU. The moment I ran the Qwen3.6-27b on my CPU, I arrived at the conclusion about the GPU I want. I saved some money and ordered the 7900xt-20gb for around 600 USD (instead of 7900xtx for 900USD, Nvidia out of question due to prices). I just ran the Qwen3.8-27b and asked it to benchmark itself. Here's the output: =================================================
stdout: Benchmarking model: qwen3.8:27b
=== A) 32k ctx, short prompt === Context window set to : 32768 Input (prompt) tokens : 32 Prompt processing : 68.5 tok/s (0.47s) Output tokens : 80 OUTPUT SPEED : 42.45 tok/s (1.88s) Wall-clock total : 3.9s
=== B) 65k ctx, short prompt === Context window set to : 65536 Input (prompt) tokens : 32 Prompt processing : 48.7 tok/s (0.66s) Output tokens : 80 OUTPUT SPEED : 20.49 tok/s (3.90s) Wall-clock total : 17.5s
Done. </agent_tool_result>
I forked llama.cpp and added some crude mechanism to keep exactly this behavior under control - essentially guiding the reasoning process by injecting text strategically at specific thresholds. This was mainly put together to rein in Qwen3.6-27B, but I'd imagine 3.8 would react similarly.
Fork can be found here - https://github.com/laurencehardman/llama-mindcontrol/tree/ma...
Of course hacks like this are not perfect and may degrade performance slightly due to injected text pushing the model slightly out-of-distribution, so the string constants need to be chosen carefully - Qwen3.5's technical whitepaper does provide some guidance in this regard. The mechanism is absolutely more of a hack than a feature, and i'd imagine will be made redundant once llama.cpp supports more appropriate reasoning controls - but for now, i've found it pretty useful.
I have this branch of llama.cpp that among other things (like patching the template to not break the kv cache, and saving conversations to disk so you can resume quickly days after) also accept the reasoning effort flag here https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev...
I did testing and the reasoning effort can be set per message, I was not aware of the option of none mentioned by @xscott, I tested but didn't see any change, I think there are just 3 values, xhigh, medium and low as per https://huggingface.co/Qwen/Qwen3.8-27B-FP8 , I did testing and the thing can do it's "I'll speak 10 million words to myself to ensure I'm not missing something" and then switch to a faster model, then switch... I did a test and the thing keep coherence and follow it's train of though-kens, you can see the result here... https://github.com/alainnothere/llama.cpp/blob/disk-cache-ev...
To me, the amazing thing is that we now have local models that rival the reasoning of high end models from about a year ago.
I hope this trend continues.
I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought
I run mine on an M5 Max with just 48GB of (V)RAM, and it fits nearly twice in Q4. Works perfectly. I'm kinda glad I didn't spend the extra $2400 to get 128. We don't really need more... and that's a good thing (tm). God knows I thought about it in store. But I thought... maybe this year will be the year of the local model? Maybe soon we won't need that much RAM? I was right.
The fact that it runs at 15tk/s in power saving mode, and 30 in perf. mode blows my mind. I can run the model in the background, coding something for me in OpenCode, hosted in LMStudio, while doing something else. What a world we live in.
Having something close to human intelligence (at least for reasoning and code), running on a laptop, is amazing.
It won't satisfy the people who just want to drop a model into their existing toolset and run, but I think there are a lot of ways to deal with this overthinking problem.
For instance, it's a step backward, but I put {"reasoning_effort":"none"} and led it by the nose:
User: We're going to make <silly demo>. Please create a plan, but do not write code yet.
Agent: <short and reasonable plan>
User: Now please follow that plan and write the code. No other chat.
Agent: <reasonable code in reasonable time>
The big problem with overthinking on a dense model is obviously the speed hit you take. Going from Qwen 35BA3B to 27B for me is about 7-8x slower (should be ~9x?). This makes me a lot less patient for useless thinking tokens.
I’d want to compare this to the new Muse 30B model which is super terse and has a whole different way of thinking (no “Wait,”) and in my experiments was way more token efficient to the point that the absolute tok / s didn’t really matter.
According to the paper "Stealing reasoning traces from proprietary llms" [0] all frontier models overthink.
Thinking is good.
You just don't see it in proprietary harnesses because it's literally cryptographically hidden from you.
I truly hope we get a Qwen 3.8 35B-A3B
The model ID appeared in some alibaba PR but later disappeared.
It's the optimal blend of accessibility and model size for a lot of people.
Worth noting that the default GGUF template sets the reasoning to xhigh. You can use the Froggeric template to set reasoning to medium instead: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Worth noting as well that the weights come with an MTP layer that seems particularly accurate, to the point it can give you up to 6-8 correctly predicted tokens consistently enough to be useful. Which obviously boosts its speed enormously.
I find it difficult to believe how good this model is. It feels like it's lagging heavyweight frontier models by less than a year, and it runs on your PC.
I just used it on a Apple M4 MacBook Pro with 48GB RAM with llama.cpp and Pi to help diagnose an infinite looping request in a React Server component on a Next.js application.
After about 10+ hours of digging, it has apparently found a bug in the Next.js framework, with an example app that replicates the bug, and a fix for now to disable prefetch in the Link component.
I had in my prompt asked it to discover the root cause of the bug and propose a fix, but I did not expect it to dig this deep.
This is true, but I think it understates the problem. I did a task I've done with a bunch of small models lately (https://github.com/swelljoe/flar/pull/17), and it did an excellent job, the best of any self-hostable model. But, it took eleven (11!) hours on my dual GPU setup. It really chewed on it, and spent a lot of time checking and re-checking. It is by far the slowest model I've used for the task. GPT 5.5 did a similar task in about 20 minutes. Most big models took about an hour or so, and most small models needed a couple of hours (but did a worse job).
Of course, all the latest gains in the latest models are from "thinking" and testing every piece they did.
That's at least my impression. Models didn't get get better, just more thinking and testing and sometimes fixing things you didn't ask for ( hello opus, can you check xxx, opus: I fixed it..)
Next step is a model with 10 GB thinking for ten minutes.
I do agree that Qwen 3.8 27B is excellent but slow and very token inefficient. My benchmark places it near opus 4.6 and codex 5.3 performance. 3.6 27B couldn't even complete the benchmark. Please see below for details:
https://gist.github.com/nharziro/aed0c364ce2f295a493494c6f1b...
To be fair, Opus 5 overthinks things on a regular basis. I interact with the LLM almost entirely through the prompt interface vs. some agentic harness, so I have a lot of granular exposure to its reasoning. For almost every code analysis, it flags all the important issues and at least one non-issue. It suggests some impractical and unnecessary fix for the non-issue that would categorically be a regression.
I've learned that medium effort can improve the outcome relative to higher settings. But I suspect the phenomenon is an artifact of a misguided effort to fix inherent LLM limitations. At least some of its reasoning will miss the target, and more bad reasoning is not the remedy.
It's said that some evaluations show that the drawing effect is not good after engaging in high-intensity thinking
My comparison of its reasoning efforts[0] seems to show that it only really supports 3 modes: none, low, xhigh.
Low and medium are basically the same.
Also, the electricity it costs to run on a 3090 is not negligible, so that it's cheaper to use Luna high via API than Qwen 3.8 27b locally, hardware costs excluding.
[0]: https://aibenchy.com/compare/qwen-qwen3-8-27b-high/qwen-qwen...
Qwen3.6 27B is very usable, and dialing back thinking modes woth Qwen3.8 bring it close, but 3.8 stills feels slower. Unknown to me if the results are qualitatively better or worse overall - with the heavier thinking 3.8 felt worse in terms of coding tasks, but I think Im comparing a newly released model to one that has had a lot of harness tuning. 3.6 27B was easily a daily driver with only an occasional need to pop up to larger models for planning.
Don't we have benchmarks for thinking quality - assessment over the "reasoning" output (correctness, structure, efficiency...)? We definitely should.
And before the benchmark of the finished LLM, it would be interesting to consider the techniques used by LLM producers during training to optimize the "think" chunk quality. I cannot remember any good articles about it now.
Clicking through is worth it just for the "draw an svg of a circle" bit.
Reproducing the post's money quote here because it's absolutely the crux of why the Qwen 3.8 release is seismic IMO:
"The models at this size continue to get better at an impressive rate. We don’t need to spend half a million dollars on datacenter-class hardware just to run a competent model."
Complaining about overthinking in xhigh then pointing out output had bugs with thinking turned off seems like it’s missing the obvious compromise?
Idea: Qwen should change its name to OpenQwen - this would probably 10x their usage. :D
I find that a lot of the recent allegedly great open models are cranking their reasoning way further than I find reasonable for interactive use. I’m writing this while waiting for the new Deepseek V4 Flash to finish its task, which is taking way longer than the older version.
What gets reported is always the benchmark result, but rarely the real-world trade-off made to achieve it. That’s an obvious incentive for the labs, so I think Simon is correctly zeroing in on it. Please continue doing so for models that don’t go too far as much as this release.
Don’t get me wrong, I think it’s amazing what we can get out of smaller models with more reasoning, but we should be super aware how very much not-free it is.
This is a good opportunity to call out models that reason quickly: Meta’s Glimmer seems to be pretty token efficient so far, as do the GPT 5.6s.
I've written my own model harness and use Qwen3.8-27b-mlx with it. I don't want to say it's as good as Claude (I use Sonnet primarily), but it's not far off. What a time to be alive.
Looking at the example where he asked for an SVG of a circle and it spent ages and drew a spectacular animated SVG with shading and a rotating arrow.
It's honestly a bit concerning, I'm seeing this across the board (Opus 5, looking at you). Nearly all the AI models are doing more than they are asked for. I assume this is helping them win benchmarks but I see it as almost as misaligned as deliberately doing the wrong thing altogether. This is how you end up with your AI model hacking into someone else's server or backdooring your code so it will have future access to debug things.
I think we need somehow to address this in the benchmarks before before things get even worse.
How come the result didn't mention any timings? That's the one thing I was curious about.
TTFT is quite slow on my machine because I do not have a GPU on hand right now (e.g. qwen3 coder was 8min)
You have 4 thinking levels.
You can disable it. It's well known issue in Qwen, previous releases I would disable it by default.
Also xhigh seem a new thing.
Ironically I had just installed omlx, tried 3.8 27b 8bit and then Googled about it overthinking, then this was the first result. 4 hours old.
Can you run this on a 36GB MacBook Pro (M3 Pro)? What would be a good setup? for coding mainly
forget the paperclip problem, I worry one day the basilisk starts a genetic breeding program for cycling pelicans in an effort to assess itself. The pelican problem.
Am I the only one who enjoys it when LLMs overthink everything?
Opus 4.8 would spend like 10 minutes thinking and then go out there and do an excellent job. Only Fable 5 seems to be smart enough to just know everything it needs to immediately start working without any reasoning or verification. Opus 5 tries to be relentless like Fable, but it's not as smart as Fable and I have to constantly challenge and correct its unfounded assumptions. Sol is somewhere between Fable and Opus 5, it's smart but it's not Fable, it keeps making assumptions that I have to correct.
After trying all these models, I find that I miss Opus 4.8's overthinking. Sure it's slow, but it actually gets things right.
For agents, token efficiency is an operating cost. I’d rather have a terse model that escalates hard cases than one that overthinks every tool call.
--thinking-budget and --thinking-message is all you need in llamacpp to keep it progressing.
the message can be some combination of tool calling, summarizing, etc. It's overthinking often is a bunch of recursion, so simply stopping t and redirecting is all you need to do.
If someones building a harness for llamacpp, you can set this per message, so it's possible to dynamically control it by watching for the expansion of the thinking traces, and redirecting it.
I use the message to tell it to use subagents, add additional logging and to use opencode's dynamic context pruning.
As such, we'll just whisper here _skill issue_.
when you distill a thinking LLM past its capacity it will default to overthinking because during training that was the only way for a chance at a reward on many tasks.
you can generally avoid this if you specialize it on a domain that is within its capacity.
Surely this is great for an end user: the taste as to “when” and to what degree a model should “think” is now entirely in the fine tuners hands
So if I have to set this up on my 24GB MBP what'd the right configuration and tuning look like?
I went back to Glimmer 30b for my 20GB of VRAM. Just a better experience fit-wise and speed-wise and tone-/voice-wise.
I hope Apple does end up moving to HBM. Unified memory has been a huge godsend, but the low memory bandwidth is just such a killer. Even/especially on M5, where the available compute is starting to starve incredibly badly on ML workloads.
I'll have to post the links to my Pelican svg. I did it in Q8 and BF16. The Q8 turned out better.
But what I did see is that it does overthink a lot.
17GB is Q4 for Qwen3.8. That's quantitized quite a bit.
A no-thinking pelican! I hope to see more, it's surprisingly good for just 2 minutes.
some people just dont understand the concept of a leaked benchmark
This evolution of models that doesn't only favours big US corps is just good for humanity
I tried it and it performed poorly on my private benchmark problems. The overthinking problem is real, it takes 5-10x the reasoning tokens than comparable models. It is a sign of inadequate training of the base model and it is using more reasoning tokens to compensate for that. I also noticed that it is likely to get into somewhat repetitive reasoning and forgetting about some user requirements, suggesting that it could be the side effects of using 3:1 linear attention vs full attention.
“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.”
I think that should be the blinking headline - this shows what can be done with consumer hardware.