alt Hacker News> If you buy your own GPU, rack it, and run an open model, there is no unit cost.
No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.
Replies
roncesvalles • today at 3:47 AM
How is this different from hosting infra or just SaaS in general?
TFA contends that there is some fundamental shift in the economics of software, but it doesn't look to be very different from either a new SaaS dependency or racking new hardware.
robocat • today at 3:15 AM
> opportunity cost
We need a better word for this because the things you didn't do are not a cost
➕ show 1 reply
These are basically all fixed costs, not unit costs. You buy the GPU once and use it for as many calls as you have traffic for, and depreciate it over a fixed lifetime. You have to power it regardless of whether it's fully utilized or not. You have to maintain it by virtue of owning it, not really based on how many queries it has served. Bandwidth is the only one that really scales as a unit cost.
Open question whether this model is actually more economical than using the cloud AI service. The whole reason the industry moved to cloud computing in the first place was because computing had very high fixed costs, and the more these could be amortized over a fully-loaded query stream, the lower the unit costs.