alt Hacker NewsQwen 3.8 Flash Next is a mixture of experts model though. It has 6B active parameters per token, so you can cache large parts of the model in RAM and move it in and out of VRAM as needed. If you do that with a dense model it gets much slower.
Qwen 3.8 Flash Next is a mixture of experts model though. It has 6B active parameters per token, so you can cache large parts of the model in RAM and move it in and out of VRAM as needed. If you do that with a dense model it gets much slower.
Experts are not typically moved between VRAM and system RAM during inference, as it would be too slow.
What happens instead is that the "dense" parts of the model like e.g. the router are put into VRAM and the remainder of VRAM is occupied by the KV cache and whatever experts still fit.
The rest - usually just the experts - reside in RAM, but it's bo big deal.
Additionally, some models utilise an n-gram table, of which only one row is read per token, so it can actually be stored on disk without issue. Example:
https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...