alt Hacker NewsFor somebody who hasn't kept up with all the different open source models and what token rates mean, how would using this compare to something like codex or claude? I assume it would be much slower and less capable..?
Replies
The model they're running is Qwen 3.8 27b. It's maybe somewhere around Opus 4.6 level, it's quite good. I didn't check the benchmarks, that's from memory/vibes.
The ~issue is though that they're using a pretty low quant, 2 bit. 4 bit are _pretty_ close to full ability, maybe degraded a bit. 2 bit is _noticably_ worse. It's about as small of a quant as you can get while still pretending it works, for this model.
This won't replace claude for you, unless you baby it a lot and are okay with it being kind or derpy.
Significantly slower and less capable than opus 5.5
However you can run comparable or better models than gpt4. Try to stay under 30b
Qwen3.8-27B is surprisingly effective, somewhere in the vicinity of Opus 4.6, but it needs to do a lot of inference-time computation. i.e. anything moderately involved requires xhigh reasoning and it will spend literally tens of thousands of tokens turning over a problem before emitting the answer. The answer/code/whatever is often surprisingly good, but if you’re in the low tens of tok/s you’ll be waiting a long time to get it.
As sibling said, you won’t hit the same quality using a 2 bit quant like this project, which is unfortunate but that’s the reality today.