alt Hacker NewsExperts are not typically moved between VRAM and system RAM during inference, as it would be too slow.
What happens instead is that the "dense" parts of the model like e.g. the router are put into VRAM and the remainder of VRAM is occupied by the KV cache and whatever experts still fit.
The rest - usually just the experts - reside in RAM, but it's bo big deal.
Additionally, some models utilise an n-gram table, of which only one row is read per token, so it can actually be stored on disk without issue. Example:
https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...
Replies
> Experts are not typically moved between VRAM and system RAM during inference, as it would be too slow.
For some definition of "typical", sure, but it depends on what performance trade someone is willing to accept. Strata LLM is geared around streaming weights as needed. It lets me run DeepSeek-4.1-Flash 1031, heavily quantized (ISTA-DASLab's 2.25 bpw quant), on a single Radeon R9700 plus 256 GB host RAM. 500 tps prefill and 30 tps decode isn't great, but it's a large fraction of what I get with Qwen3.8-27B quantized to fit entirely in VRAM for short prompts, it's faster past about 80k context, and DS4.1F can use a 1M-token context. It takes forever to fill that, or even to refill if the KV cache doesn't match, but it works.
> and the remainder of VRAM is occupied by the KV cache and whatever experts still fit.
Those days engines try to optimize "whatever experts" part and can use remaining VRAM as cache, moving hot experts to GPU during inference (but less used experts keep executing on CPU, yea).
e.g. llama.cpp recently added flag for that: https://github.com/ggml-org/llama.cpp/pull/29887