alt Hacker News“The fact that a 17GB file can do all of this stuff on my home machines is a miracle. Once again, I’m delighted and amazed at how much progress local models have made this year.”
I think that should be the blinking headline - this shows what can be done with consumer hardware.
Replies
Like the old proverb: "The marvel is not that the bear dances well, but that the bear dances at all."
Full agree. I until very recently thought AI tools of today were limited to prohibitively expensive high end hardware hosted in data centers.
I was surprised and amazed to get "decent" (with the expectations set right / low) coding performance out of Qwen3.5-9B on a decidedly medium end Radeon 9070 paired with a 5700x3d and 32GB of DDR4 RAM.
We can finally reason with and "talk" to our hardware.
The other implication here is that this is all software improvements and optimization. There might be a lot more wiggle room for improving quality over time. It seems the model and reasoning quality is improving faster than the hardware currently.
The over reasoning that Simon Willison highlights here is a real issue though. I've observed it with some of the OpenAI models as well. They are prone to overthinking and overengineering things.
What I would love is models that figure out their own appropriate reasoning effort given a task. I'm spending too much brain cycles worrying on what model speed, reasoning, and quality settings to pick. It's not just a cost concern it's also a time concern. Wasting a lot of time for simple UI tweaks because the model is set to high or ultra or whatever is counter productive. The last few iterations of frontier models seem to emphasize benchmarks and reasoning effort.
But of course the day to day reality of many developers is that they are trying to solve relatively simple problems compared to e.g. proving some so far unproven theorems, solving some Nobel prize level problems, etc. I'd love my tools to start making sane choices based on what I ask rather than defaulting to "boil the oceans". These tools need some kind of Auto select. Mostly Ultra is overkill and a waste of time and resources. And of course with local models, keeping simple things local is a nice option.
It's nice to have Sol Ultra extra fast as an option in my back pocket. But it's complete overkill 99% of the time. And it's not like most users make good choices here or are even capable of making good, informed choices. The models are more intelligent than the tool UX. Arguably, a local model of very modest size might be able to do better for this specific choice.
Tried yesterday on my own laptop (a UltraCore 7 255H without dedicated GPU,with 32 GB RAM), it wasn't even starting thinking, even on a small context window (65k)
In this kind of moment, I really wished hardware manufacturing and demand situation is in much state. Imagine this can be accessible by everyday people with only 6 months hardware market gap. The societal impact would be much bigger.
I wish we could have better hardware and I think the tech is there for a few years already.
I've gone in (too many) details last night with the calcs: https://news.ycombinator.com/item?id=49324600
I thought the same thing, and I generally do a lot of animation in my work, and the results in motion graphics with Qwen are impressive, I really fell in love with it
For me that moment was Gemma 4 12B QAT. You're not suddenly going to start throwing your hardest programming problems at Gemma 4 12B QAT, it is still 15B parameters less. It's more that, aside from pelican art which isn't what local models are for, I didn't see anything on Simon's post that it couldn't assist with or largely succeed at.
It can run 80-100t/s on a laptop, can understand images natively and do bounding boxes, read tiny text, understands audio natively as well and can transcribe or translate anything you say, can do accurate long context retrieval with pretty large context windows, tool calling, excellent reasoning and is very token efficient.
It's only 7GB including the mmproj or 8GB with MTP. The Qwen 3.8 27B model Simon was using is ~18GB with MTP+mmproj, rather than 17GB alone. The point is not really that you compare these models directly, but that Gemma 4 12B QAT was really a special moment in model releases deserving of a similar reaction relative to its size, but was mutilated by Google themselves, Unsloth and Llama.cpp.
The overall appreciation I think we're seeing this year in particular is that people are easily surprised when multiple things are improving simultaneously which produce seemingly exponential changes. It isn't just that models are getting smaller, or that reasoning is getting better, or that speculative decoding is becoming mainstream, or that models can understand audio and images better now, or that they can reliably call tools which expands their capabilities, or that context windows are getting larger, or that accurate retrieval is improved, or that.... and so on. It's all of them narrowing in at once that is starting to make local models incredible and truly useful for far more use cases on the existing hardware people already have.