Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one.

It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.

I mostly agree with the prediction though maybe a bit more pessimistic about the timeline. Also I'm not sure our current usage of parameter count would make sense in this scenario, such a feat would require compressing current parameters in a manner much different then something producing a bit count per parameter. A hypothetical example would be using a single seed parameter per layer which then passed into a noise function produces the functional weights for that layer, able to reduce per weight size to sub bit levels (256 bit seed, producing 16K weights).

> on a decent speed

But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

That is awesome!

I am curious about the decision to not use GPU since this is for Apple Silicon.

Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?

> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.

This prediction alone isn’t useful at all without a bound on speed and maybe quantization.

You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.

If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.

> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.

You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.

You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.

Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.

Downloading now just 'cause the repo name

How are the thermals? I noticed that running any serious workload locally heats system fast.

> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat

It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)

Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!

By early 2028, major players like Intel, AMD, QC will ship accelerators in consumer laptops capable of running ~1T MoE models at ~100 tok/s

I tried Qwen3.6-35B-A3B, but it couldn't generate a 50-100 line Clojure file without having broken parens mismatches. I know Clojure isn't super popular, but the syntax is pretty simple and the frontier models do fine with it.

> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second.

That is no where near decent at all.

[dead]