alt Hacker NewsEveryone keeps repeating this who doesn’t understand the underlying technology.
Small llms are still way more efficiently server on big GPUs.
Sharing server capacity takes advantage of the massive parallel throughput and sharing of memory bandwidth.
You are sharing the GPUs with thousands of concurrent users.
Replies
It may not make financial sense for someone retired, not into tech, and/or data privacy to host their own LLMs. However if usage of AI in day to day lives continues to increase, I think it will eventually make sense for the majority.
Many tasks suited for AI assistants are background asynchronous tasks. They can run in the downtime where immediate demand is low, keeping overall utilization high enough.
Your argument is similar to those who argue that owning a GPU for gaming doesn't make sense when you can stream from something like GeForce Now. However like with gaming locally (improved latency) there are also benefits to local AI (data privacy).
The power of small models isn't only that you can run them on local hardware. You can also fully own your data and workflow, and choose/fine-tune models for your specific use-case.
for clarity, I'm not agreeing with GP that small models will mean doom for data center projects
> Small llms are still way more efficiently server on big GPUs.
Yes, but the privacy aspect means that for many, many applications slower local will still be preferable to faster remote so long as the actual model performance is the same.
True, but local setups can run LLM requests in parallel too. In this case efficiency gap is much narrower.
Indeed. Local compute and RAM are a some of the most wasted resource, sitting idle 95 percent of their life. Datacenters flip that ratio.
I’m sure you could find this exact same comment regarding technology in 1999.
FWIW it is entirely possible to square the notion that small models will still be hosted on cloud hardware with the idea that the data centre buildout will end in tears.
Many analysts (and Microsoft) think even now that if everything committed gets built there will be considerable oversupply and there is not the revenue to pay for it.
If small models do continue to improve in unusual ways (I think there are limits) then the marginal need for cloud AI compute could fall precipitously beyond current estimates. The marginal need for consumer AI could almost totally collapse if someone makes good progress on very small reasoning and tool-calling models (which is a modestly big if)
The possibility of the data centre boom resembling the Chinese real estate bubble is not inconsiderable.