alt Hacker NewsCan you host the model for a lower cost per token than you'd pay Anthropic or OpenAI for a similar level of intelligence? I doubt you're beating their efficiencies of scale.
Replies
criley2 • today at 7:01 PM
No, and the reason is simple: Usage is bursty and if you don't maximize usage of the hardware you're going to lose on price.
Ok you can host this model once. What if I want a dozen subagents? Ok you can host it 12 times at once. What if we go a whole week only using max 4 at a time? Etc etc. The limits imposed by self-hosting might be bearable for a variety of reasons, but it's going to be more expensive and less convenient/useful.
I dont have estimates on the cost of running models, but I think openai and anthropic are running on subsidized prices. At actual prices it might be worth it in the future.