alt Hacker News> and the remainder of VRAM is occupied by the KV cache and whatever experts still fit.
Those days engines try to optimize "whatever experts" part and can use remaining VRAM as cache, moving hot experts to GPU during inference (but less used experts keep executing on CPU, yea).
e.g. llama.cpp recently added flag for that: https://github.com/ggml-org/llama.cpp/pull/29887
I was not aware of that, so I downloaded the latest release (which includes but doesn't document this flag), but performance unfortunately tanked in my case.
Even If I set the fit target to a value which without the cache would leave that VRAM unused, I get about a third of my usual token rate.