alt Hacker NewsDon't waste time on bonsai unless you have like 16gb or less ram (ram, not vram). Although I'm pretty sure you could still run this model well on 16+8, and qwen3.6 35b-a3b runs well on ~32gb combined ram at q4. Just offload most of the experts to cpu/ram if you have limited vram budget. With the moe models it doesn't hurt performance too much.
But bonsai is basically trash on real agentic coding work. I've tried both bonsai models, all the quants, messed with the hyperparameters, tried to fine tune (didn't succeed), and it just kept looping or failing to finish tasks. Qwen3.6 35b-a3b is my current go to for agentic work for any systems under 24gb vram, although honestly I'm not sure the dense qwen3.8 27b is meaningfully better. The main difference I see is it does a lot more testing attempts and burns a ton of tokens + compactions, so every task it takes 1.2-2x as long as 35b-a3b does. But the quality of the results is comparable.
Replies
In my experience, Qwen3.8-27b is so much more capable than its MoE predecessors. I find it odd that you think it produces the same results.
Thank you, very interesting. I also often use qwen 35b-a3b, when doing things locally, as I only have ~4.5GB free VRAM. My other goto is gemma 4 26b-a4b, which has faster prompt processing and the same token generation speed.
I've experimented with kolibri1 (78B-a3.5b), but despite the similar size in active parameters it is much slower. Even though the experts fit completely in main memory, as for the other models. I haven't tuned llama.cpp for it as much, so maybe there is still something to gain?
I love your detailed explanation of things that I had little to no knowledge of, but I’m also stuck wondering if I’m using my setup correctly, as I have 16 GB of VRAM available but also 64 GB of regular RAM in my computer. Every time I use some setup or some configuration that overlaps or overflows between the two RAMs, my token per second drops significantly, me thinking that it is not set up correctly, so I’m stuck with 9 billion parameter models myself, even though I was under the impression with 64 GB of memory that I would be able to run something like a 30 billion model.
I do run the gpt-oss-20B fast on my 16GB vram, but it’s, limited.