alt Hacker NewsQwen3.8-flash-next starts being "functional" for me above 4 bits weights and int8 kv. Preferably 4.5 bits.
Likewise those 20t/s they talk about are often at very short prompts and use very favourable MTP setup. The real number (on prose etc, where mtp is less useful) is in the teens.
Even 20t/s are below my usability requirement for an interactive model, but if that is all you have it is brilliant.
However if you want to use it for work similar to what people use 90eur/month claude code subscription for you need a mashine like what I have. Ryzen 7950x 192GB RAM and 6x rtx3090.
Cost today around $20k.
If I was building it today I might go for xeon ddr3 platform for cheaper ram, higher total ram bandwidth and more pcie lanes, but i use what I already have.
So I have this and I still pay for 90eur/month claude code subscriptions and $30-50 a month for open source cloud models. Why? Because if I wanted to do similar amount of work I do with claude code I'd need at least 2 if not 3 such mashines and this being winter (crappy solar output) my electricity bill would be over $1000 a month.
I once calculated my claude code use would be worth thousands of $ per week if I paid api prices. Simply speaking Anthropic subsidises these subscriptions a lot. Why would I not use it?
Of course all the disadvantages of cloud and opaque hosting have hit me many times, but free money is free money.
Replies
It depends on your workload and use case IMO. If you work at a company that is using LLM coding as staff augmentation (my current and last job) a subscription setup and (usually) buying tokens is cheaper. As someone working on a side project I’ve been getting a really good ROI on my box made from used parts running Qwen models. I need something that will make progress but I’m sending a prompt, going about my day/dayjob, and check back in.
There’s no way I’ll beat a $200 subscription on total output but I am too busy piloting LLMs, taking care of my kid, having a social life, etc to max out a cloud subscription. I can however blow through the lower tiers of cloud subscriptions when doing hobby work.
A bunch of what I’m doing these days is in the category of people doing porting of ancient software or unexpected software. It’s fun and every so often I can contribute stuff back to the community. The box runs at under 1kW and I have the ability to remote manage it to turn it on and off when I’m not using it. Given it’s winter it’s a slightly annoying sounding but equally useful space heater in some ways .
With current software you don't need those 6x RTX3090 anymore. With strata you get >50 t/s on Q4 on a single 5070Ti. And with 192GB RAM you can run Q6 without reducing speed too much.
With 6x3090 I'd most likely go for even bigger - DeepSeek?
I take issue with your numbers. A $20k system and $1000/month in power are way off. Like both an order of magnitude off.
> Likewise those 20t/s they talk about are often at very short prompts and use very favourable MTP setup
It’s not “they” talking about it. I am personally running the machine I described, and ~20t/s is the average of what I get at every point in 128k context for coding tasks. Dense models like qwen3.8-27b slow down at longer contexts, but flash-next does not. It’s slower for prose, but I don’t really do much of that. Yes it’s barely usable speeds - i’ll be adding a better gpu in a day or two when it arrives.
To run 4bit quant and 256k context, yes you’re going to have to spend more than $600. But not $20k. I have friends running systems that cost less than $2k getting >80t/s output and >1500t/s prompt processing, at 256k context, 4bit quant. A 5060ti with 16gb vram is all you really need to add to this config to get into that performance range.
I live somewhere with cheap power, which helps - but we’re talking about maybe 250W peak power consumption (180W for the gpu, the rest for the system). So if you ran that full tilt 24/7 it’d be ~6kwh per day.
Assuming you have very high power costs, that could work out to something crazy like $3/day, $90/month. I have no idea how you imagine getting to $1000/month without running a fleet of machines like this cranking out tokens 100% of the time.