alt Hacker NewsI mean, I might be missing something, but isn't part of the idea with cloud infra that you can scale down as well?
Large orgs with significant demand might go out and buy local LLM hardware, but most businesses probably don't want to bother dropping $2k on a box with a beefy GPU and would rather just pay the lowest subscription tier so their employees can occasionally make queries.
Plus, you know, the whole economies of scale thing. Local LLM has a lot of privacy and independence benefits, but I'm not really seeing the world where it becomes more energy- or cost-efficient to buy your own hardware (and use it 1% of the time) versus sharing a giant machine, or even the same machine, in a datacenter (where it has a much higher utilization factor).
Replies
All white collar work will be done by AI soon, there won’t be scaling down, just scaling up. I do food trucks and festivals and I’ve got AI doing so much of my non-meatspace work now that I’m buying $10-$20 a day in tokens. At that rate, hardware starts to look cheap, and I have less of a use case for it than most white collar workers.
I'm pretty sure a small-to-medium org has plenty of things that could be queued/scheduled to run when there is downtime. It requires some planning and thought though and I don't think most orgs are there yet.
$2k? LOL. That's not even the GPU budget, these days.
The economies are currently out of whack because of underproduction of components and memory, so LLM providers have a few years of runway to entrench. Plus, the whole capex Vs opex thing that helped AWS will help here too, for sure.
At some point, though, things will change. More production will come online, and providers will have to end the current speculative subsidizing and jack up prices.
It's a bit like the dot-com era: the initial rush to land-grab web portals and e-commerce sites eventually died, once enough skills and infrastructure came online, and the bubble burst.
> isn't part of the idea with cloud infra that you can scale down as well?
Yes though anyone with an aws bill knows that’s sales pitch lies
The bill only goes one direction and it ain’t down