logoalt Hacker News

kgeist • today at 8:18 PM • 0 replies • view on HN

It says 46 tok/sec for Qwen3.6-27b 4 bit on RTX5090, but I get 60-70 tok/sec on a small context like that (5 bit weights with 8 bit KV cache), although it does slow down to 40 tok/sec on large contexts. If I add ngram speculation, then it can increase to 100+ tok/sec on code.

There are many knobs to tweak to make it faster, which the site ignores.