alt Hacker NewsI would be very interested in a similar benchmark for *KV cache* quantizations.
I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there.
I also heard that KV cache quantization matters more with longer contexts. It may be interesting to benchmark this too: what the quality looks like on different combinations of model quantization × KV cache quantization × context size.
Replies
On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different.
I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use of OS.
my coding agents regularly reach 200k context used without noticeable degradation.
P.S. I used setup from: https://github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2
You don't have to quantize all layers and all dimensions uniformly, FWIW
There was a study specifically related to Qwen3.8 27B that showed that kv cache quantization has almost no impact on this model all the way to q4:
https://arxiv.org/html/2609.04098