alt Hacker NewsI'm trying and failing to find value running a potential Qwen 3.8 27b dense model on a 16 core, 128 GB of ram, 2080ti box. Yes, the GPU yells for help, but the problem is that no math works to upgrade this machine even when pouring $200 in rent every month into the large model providers...
How are you all justifying economical use of these local models right now? What's the cost efficient way to do this and do better (even with models evolving over time and losing now vs later) than the big labs?
Replies
There's no point, except if you want privacy and independence. I am playing with a personal assistant that checks my emails, calendars, sends me an agenda and maintains my TODO list. I am not sending such data outside.
Running locally for me is mainly about learning, maintaining control+privacy, and helping shift my coding+design process to leverage LLMs. I guess if you made me boil it down to a single word to justify the cost I would just say: tuition.
Sounds like we have similar boxes - mine has a 10 core CPU, 64 GB of ram, and a 2070 Super. My motherboard had two unused PCIe3x8 slots and doesn't support Blackwell GPUs. I bought a couple of brand new Ada generation RTX 2000s with 16GB of memory for under $1400 to get to 40 GB of VRAM. That will easily run Qwen3.6-27b at a 6-bit quantization and 80,000 token context size. It isn't fast (19-21 t/s), but using pi-coding-agent is fine.
Now, my instinct is that I am giving up SOTA performance on agentic coding with this setup and LLM. But the gap between my setup and SOTA commercial models is small enough that it doesn't matter to me.
The only scenario is if you have enough work to do batch inference. Using a tiny fraction of GPU capacity to decode a single request at a time just doesn't make sense, as you say.
If you can do inference on the CPU, drop the GPU : it should be faster.
I use opencode go and pay 10$ a month. Works good for me.
> $200 in rent every month into the large model providers
We all know that is hugely subsidized, and I guarantee that OpenAI and Anthropic are looking to enshittify that ASAP.
The enterprise users, however, are not subsidized like that. They pay per token. And some developers in those companies are chewing down a lot of tokens. Self-hosting an open weight model could be a massive savings very quickly. It also gives them negotiation leverage when talking to OpenAI and Anthropic.
You will simply not get more value out of running a local model vs paying for a subscription/API from the cloud in 2026. There is no math that will make local models come out ahead in $/intelligence/token.*
The point of local models is privacy, offline use, and maybe no guard rails.
* Not talking about enterprises that buy DGX racks and host Chinese models for internal use.