alt Hacker NewsWorking on a blog post to document it all, but that will take a few more days. The short version: the big rig is probably $25K, ttft is dependent on how much prefill you have but typically a couple of seconds (2K prefill/sec in one setup, about 500 in the other), tgs 100 to 150 or so depending on what it is doing and how well the speculative decoder works with the current context. Electricity is pretty bad, full tilt it's about 2500W or so but I've power limited it to 1500 to keep it civil. It is not cheaper than using a frontier lab (or at least, I don't think it is), but the independence bit is the important part for me and I don't care if that carries a surcharge, even if it is a substantial one.
Replies
The benefit of the agent working firstly for you rather than for Anthropic or OpenAI will become increasingly apparent. We're already seeing signs: remember when Anthropic openly promised to have Claude sabotage people using it to create competitors? And OpenAI is looking for revenue opportunities from advertising, which can't end well for their users.
Just think of how user-hostile other online services have gotten, and imagine how acutely painful the proprietary LLM providers could make theirs. Thankfully, open models provide some counterbalance.
Would you use hosted models if they promised anonymity and no log retention and servers were physically located inside jurisdictions that respect privacy (Switzerland, Norway or similar, I'm not into law) if that meant better price/performance ratio than your current rig?
> It is not cheaper than using a frontier lab
Are you comparing to subscription prices or API prices? Because if it's the former it won't last forever.
Did you get insurance on all this stuff? I imagine home insurance would not qualify for a bespoke home-datacenter.
I'm not saying your approach is wrong - but for my workloads, it makes a lot more sense to just pay for a decade (!!!) of Claude Code 20x Max.