alt Hacker News> Experts are not typically moved between VRAM and system RAM during inference, as it would be too slow.
For some definition of "typical", sure, but it depends on what performance trade someone is willing to accept. Strata LLM is geared around streaming weights as needed. It lets me run DeepSeek-4.1-Flash 1031, heavily quantized (ISTA-DASLab's 2.25 bpw quant), on a single Radeon R9700 plus 256 GB host RAM. 500 tps prefill and 30 tps decode isn't great, but it's a large fraction of what I get with Qwen3.8-27B quantized to fit entirely in VRAM for short prompts, it's faster past about 80k context, and DS4.1F can use a 1M-token context. It takes forever to fill that, or even to refill if the KV cache doesn't match, but it works.
Replies
Something doesn't seem right. Vanilla llama.cpp can hit 50 t/s on an R9700. Q6 (albeit with limited context of 64k, more if kv cache is quantized) fits entirely on the card.
I've tried llama.cpp's new --moe-cache-mib flag and it lowers performance to about a third of what it is without the flag.
And no wonder really - PCIe 4.0 x16 has a one direction bandwidth of around 31.5GB/s - much slower than system RAM - in my case about a third the speed and I think it's no coincidence that I'm getting this exact rate of slowdown.