alt Hacker NewsI love your detailed explanation of things that I had little to no knowledge of, but I’m also stuck wondering if I’m using my setup correctly, as I have 16 GB of VRAM available but also 64 GB of regular RAM in my computer. Every time I use some setup or some configuration that overlaps or overflows between the two RAMs, my token per second drops significantly, me thinking that it is not set up correctly, so I’m stuck with 9 billion parameter models myself, even though I was under the impression with 64 GB of memory that I would be able to run something like a 30 billion model.
I do run the gpt-oss-20B fast on my 16GB vram, but it’s, limited.
Oh you should be running flash next with strata. Just go to stratas github page [0] and it will explain everything. It runs quite fast on 16+64, you should be able to get 50 tok/s or so. It's a 180b model and it feels a lot smarter than most of the other open weight options. As far as knowing "what will fit" in general that's hard to know these days. The moe models are hard to predict and run wildly different in different engines (llama.cpp, vllm, strata likes, etc) but they tend to be the only good options around 16gb vram. Dense models like qwen3.8 27b are only good if they're entirely in vram. Moe models you need the active count + a few experts (although it will run faster the more experts you can fit).
[0] https://github.com/Niko1221/Strata