alt Hacker NewsWhy would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.
Replies
It's slower than the 4:1 ratio would imply, but it does indeed work.
Things get really slow if the model doesn't for in vram + ram and you have to go from disk to ram to vram.
5090 can do all but lossless NVFP4 (OMMA) and NVIDIA does fairly good quants of most anything popular. This isn't quite a 4x reduction from what you see on the label because they're a little conservative with the QKV projections (IMHO) but it's on the order of that. So a dense model at 50-70B parameters is the sweet spot. It's a great card for strong dense models.
In principle you could have bidirectional PCIe x16 pipelining at it would move the roofline a little with fast DDR5, I think llama.cpp has a flag for it.
Or go rent a B200 on vast.ai for 4 bucks an hour or thereabouts, a single heavy Opus session for a couple hours is like a week of any model on vast or RunPods.
NVIDIA publishes something called NGC containers that generally work out of the box. I started running Qwen3.6-NVFP4-MTP locally and then I'll put something heavy on Baseten if I'm lazy or Vast if I want a good deal.
Opus (and maybe now 5.6) are still the strongest for like, the really delicate shit, kernel modules or something, but that's on pace to cross over this year, and the overtraining and misalignment are getting so bad when they phase 4.6 out I'm pulling my plan. I don't pay to get gaslit about Constitutional AI.
It's time to have an exit strategy.
There are two phases to LLMs:
1) prefill
2) decode
For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU.
For decode, you are bandwidth bound, and it is difficult to batch multiple output tokens together. There is no benefit to sending your weights to the GPU because even if it internally has insane bandwidth, you are still bottlenecked by system RAM (and adding a pcie5 upload would bottleneck it further). This is the number people usually talk about when they say they are getting a certain tk/s.