Considering how all the big players are playing fast [1] and loose [2] with limits, billing [3] and adding undisclosed changes that burn your tokens on autopilot [4], it can't happen soon enough.

[1]: Limits may change without notice, including due to capacity constraints. - https://support.google.com/gemini/answer/16275805?sjid=14713....

[2]: "standard limits" are never defined - https://support.google.com/gemini/answer/16275805?sjid=14713...

[3]: https://tobyonfitnesstech.com/blog/anthropic-refund-scam/

[4]: https://news.ycombinator.com/item?id=48947776

> at best it seems like we’re heading back to the “server under your desk” era of IT again

Maybe in the very long term. If companies go local, the efficient model is to buy some big hardware to share among developers.

I run local models. Even with 128GB unified memory systems or a 5090 or RTX 6000, the generation speeds X model quality X context length is still far behind what I get from my SOTA model subscriptions. I also pay a lot more for the locally generated tokens in electricity and hardware costs. I'm also limited in parallel requests to the local box. The list goes on.

I really like running local models, but for any given point in time it's more efficient to have a big central box aggregating requests and churning through them. So maybe companies buy $300K servers and try to split it among 30 users instead of trying to buy 30 x $10K boxes.

More likely, they rent time on cloud servers by the month so they can adapt the hardware when new models come out with new requirements.

Then some day in the distant future when hardware is cheap and plentiful again, it might make sense for us to go back to individual boxes under the desk.

Why do you say API based llms looking iffy at best? Do you just mean current profitability due to market pressures from some companies’ subsidized investor money?

Surely, even if you’re just using open weights models, it should theoretically be cheaper to use them in a highly optimized cloud architecture(even with vendor markups) rather than each person serving their own models from much less efficient (and more importantly, much less consistent volume) self-owned “server under your desk”?

I've been coding using the LLM server in my living room for the past few weeks, and I haven't had this much fun with tech for ages

That’s part of it. There’s also just a natural back-and-forth between what I call “time sharing” and “personal.” When the thing you want is expensive, you share it remotely, but as soon as costs fall, everyone wants it under their desk.

more likely that hosting and delivering the models will be commoditized, much like how DO, Linode, Hertzer etc all commoditized VPSs and server hosting. And you'll end up paying for virtual hardware size (or compute resources) rather than tokens

What do you mean iffy? The major AI labs are gross profitable when selling access to inference. In addition, the best models have trillions of parameters and are most efficiently served on large, expensive clusters and served to many concurrent users.

This release is not a meaningful improvement in any metric over 5 months old Qwen 3.6.

DS v4 Flash update maybe, but it is too big for typical Joe's desktop.

I wouldn't say "server under your desk", necessarily; more of an "Linux getting big" era of IT.

If you want to host the model on the server under your desk, you can. If you want to build a data center on-prem to host it, you can. If you want to pay a cloud provider to host it at their data center until you figure out how to scale it without their help, you can. It's like when people were first building commercial services to support Linux-based OSes, and people were also still hacking on it on local machines.

APIs may still have their place - maybe you just want to throw your devs a known quantity with all of the management built in - but it's not going to make Sam Altman a trillionaire, which is something anyone outside of the SV echo chamber could have figured out as soon as the first real competition to OpenAI emerged.

With how expensive consumer hardware is and will continue getting (due to LLM demand), good luck getting a "server under your desk" for something less than an arm, leg, and first born.

Until A100 prices are reliably under 1.70$ an hour, there is no GPU/AI bubble and Michael Burry doesn't know anything about GPUs.