alt Hacker NewsThank you, very interesting. I also often use qwen 35b-a3b, when doing things locally, as I only have ~4.5GB free VRAM. My other goto is gemma 4 26b-a4b, which has faster prompt processing and the same token generation speed.
I've experimented with kolibri1 (78B-a3.5b), but despite the similar size in active parameters it is much slower. Even though the experts fit completely in main memory, as for the other models. I haven't tuned llama.cpp for it as much, so maybe there is still something to gain?
I can attest to the fact that Bonsai indeed tends to loop and it can be something really simple, like my usual test:
> What are signals in Angular?
> How does that relate to RxJS?
This was enough to loop Bonsai 2.