alt Hacker NewsI've been wanting to run open weight models lately to give them a shot with OpenCode. However, I get the impression that models like Qwen and Kimi k3 are impossible to run locally? I have a RTX 5090 and 64 GB of RAM but the models seem to be much larger than that. What's the route to start using these models? Bedrock?
Replies
OpenRouter is (roughly) a single proxy between you + many different models + providers. it works with opencode (+ many other products), and is relatively convenient for trying out a bunch of models.
for example, they already have qwen3.8-max
https://openrouter.ai/discover?model=qwen/qwen3.8-max
note that they add some fee ontop of things (maybe 10% of spend?). it isn't htat big of a deal for general experimentation, but if you end up wanting to use a single model in a higher-volume way, it likely makes sense to cut them out of your stack.
Bedrock seems to have stopped adding new open-weights models, and mostly only has Anthropic and OpenAI stuff now. You can get Qwen 3.8 directly from Alibaba: https://www.qwencloud.com (proprietary variant) or from DigitalOcean (this variant, probably also from others soon).
On your 5090 you could easily run a smaller model like Qwen 3.6 27B: https://huggingface.co/collections/Qwen/qwen36 or Gemma 4 etc., or as mentioned there's a Qwen 3.8 27B coming out in a few days.
You could easily run any of their 30B-or-less models which is what most people are waiting for.
Apparently the ~30B variant will be released on Friday?
Fireworks or OpenCode Go
5090 is plenty for the Q4_K_M quantized version of 3.6 27B with reduced context size.
I run it on a 3090(24GB) and 64k context using GGUF format and llama-cpp. Double 3090 gives you 128k, quad 3090 gets you to full context - 256k.