alt Hacker NewsWhat configuration are you using? On both vllm and llama-cpp, I get significantly higher speeds from gemma4 than qwen3.6 (with their respective speculative decoding methods).
Output TPS in vllm for instance:
- Gemma4 26B-A4B: 200-300TPS
- Qwen3.6 35B-A3B: 120-180TPS
- Gemma4 31B: 80-120TPS
- Qwen3.6 27B: 60-80TPS
This is for a first request on a dual 5090 setup, with their respective speculative decoding methods.
Replies
Dual 4090, getting 85-113 t/s depending on task (draft seems to speed up quite a lot, disproportionately more for content like svg etc):
./llama.cpp/llama-server \
-hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL \
--webui-mcp-proxy \
--no-mmproj \
--parallel 1 \
--kv-unified \
--flash-attn on \
--fit off \
--split-mode tensor \
-ngl 999 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
-ub 256 \
--no-context-shift \
--host 0.0.0.0 \
--tools all \
--jinja \
--ctx-size 262144 \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--reasoning on \
--chat-template-kwargs '{"reasoning_effort":"medium"}' \
--reasoning-preserve \
--temp 1.0 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0
IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.
Have you tried running it on a single 5090? Dual 5090 require https://github.com/aikitoria/open-gpu-kernel-modules for higher perf. Are you using TP?
Single 3090 under llama.cpp:
> with their respective speculative decoding methodsYou're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.