alt Hacker NewsAs someone who has long bought into the idea that the labs are not profitable on inference, I have a question. How is it simultaneously true that there is more demand for inference than supply (apparently by orders of magnitude), but also that inference is so cheap that labs can make healthy margins? It seems like the inference providers, not the model providers, should have the leverage. Why haven’t they said, “give me a bigger cut of each token or I won’t serve your model”?
Also, what happens when the inference providers suffocate on their own debt burdens? Who will serve inference?
It's a few things I assume:
1. Inference providers are largely a commodity business. While demand outstrips supply, there are many providers of that supply, which does somewhat constrain how much leverage a provider has.
2. The models are not interchangeable as of yet (though this seems to be changing). A lot of people want the latest Anthropic or OpenAI model; as such, the labs still have considerable leverage.
3. A lot of the inference providers have used their leverage by doing equity deals with the labs, agreements to have some of the deal revenue spent on compute, etc.