alt Hacker NewsYou are mistaken. I'm running Qwen 3.7 28B 4bit (MLX) with a 200k context window and everything total is 32GB RSS.
Is this the best? No. That's why I said the sweet spot. Getting from 16GB macs to 32GB is perhaps possible. Jumping to 64GB or 128GB as the default is simply unreasonable right now.
Replies
Memory used : 38GB, and I haven't even started a LLM nor podman, I always fight with memory when using LLM on my mac with 48gb.
And I don't remember to have been able to have pushed to 200k context Qwen 3.6. 3.8 is running on my RTX 5090.
I assume you mean Qwen 3.8-27B? Yes, you can run this in 32GB of RAM, but it's very context limited. With KV cache compression and other techniques, it's better now than in the past, but I'd still want more RAM, personally.
EDIT to add that you need to reserve 8GB for the system if you don't want to cause problems on macOS, which means 32GB RAM = 24GB max for model + context. It takes 18-19GB to load a 4-bit quant of Qwen3.8-27B, so I'd be really surprised if you can actually get a 200k context window. You need to fit within a 24GB WSS (which is generally a more constrained RSS) to get stable performance on 32GB RAM.
I have a similar machine, and briefly poked at running a local LLM, but got discouraged after a couple days. The quality, responsiveness, and impact on the rest of the system didn’t seem worth it to me.
What sorts of things are you doing with the local LLM? Anything interactive? Should I take another look?