alt Hacker News128k context window is a complete non-started for us. We need to optimize our most needy agentic jobs, but our average context is well above that
Replies
dofm • yesterday at 7:41 PM
The model card does say 131K+ but I have no idea what scope the + really has in practice. Feels like overclocking; you're-on-your-own territory?
One thing I will say is that its thinking traces are really quite terse. It genuinely seems to spend many fewer tokens on reasoning. So that might help a bit.
I was able to use rope/yarn scaling with llama.cpp to extend the context window to 256K and it seems to be pretty usable on the debugging-and-bugfixing session I have that’s 216K tokens deep. No infinite output loops, reasoning is still coherent, tool calls appear to be passing and failing at roughly the same rate as a fresh context window. Haven’t tried going to 512K or higher yet but 256K definitely seems usable to me on a single Radeon R9700.
I added these arguments to my llama-server call, using the official GGUF release from Meta’s account on HF.
`--rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 131072 --override-kv muse-glimmer.context_length=int:262144,dflash.context_length=int:262144`