alt Hacker NewsI have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?
Replies
kamranjon • today at 12:13 AM
I am running 3.8 27b at q6 quant with 160k context on a 32gb video card (arc b70 pro) - I quantized the kv cache at q8 - that is the only trick really - works great.
ig0r0 • yesterday at 8:48 PM
yes, with qwen3.8-27b-4bit run via rapid-mlx i can get to about 200k
➕ show 1 reply
pram • yesterday at 9:51 PM
You should try Glimmer MTP. Qwen3.8 27B seems to have weird memory and caching issues on oMLX
carloslfu • yesterday at 10:30 PM
[dead]
We are running 35b-A3b with 264k context (the model's default max) using vllm and the "frog" jinja templates: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates and had good luck. We are mostly running agentic workloads though, rather than coding. 27b has a slightly higher agentic job completion rate (95% vs 92%) but the 3% trade off is worth it because the A3B is sooooo much faster, and we reprocess the other jobs with a different model. Don't sleep on the froggeric templates.
Qwen: Looking at you for a new ~35B MoE! Please and thank you