alt Hacker NewsOpen models are probably also comparatively astronomically expensive to train - just less so than the frontier models because they’re somewhat smaller, +/- the creators are more incentivised to focus on getting more from less compute because they’re have to, +/- they rely on distillation of the frontier models and this is more efficient.
But efficiencies aside; creation of open models still requires a lot of money and compute from a large organisation which is willing to accept zero return for that spend. This largesse is unlikely to continue forever; so the question is which will crack first, the frontier models’ business model or the fast followers’ generosity?
Replies
I don't think it needs to be framed purely as generosity. You just need a sufficiently self-interested actor that sees open ecosystems as a necessary part of reducing their own risk profile, relative to the alternative of complete reliance of a third-party business that can take an exorbitant cut and/or Sherlock them at any time.
Valve and SteamOS are a good example of what this idea looks like in practice. (Though they may also illustrate a third thing you need: a privately-run company, that has enough profit, and enough commitment from leadership to the company's vision, that they can make long-term bets without having to eventually bow to investors seeking short-term gains.)
I’m not exactly sure on the “how” but it only makes logical sense for (non-AI) companies to band together to fund the training of a shared model. Apple is a great example, AI is not their core business but they still require it.
The only thing that took us down a different path is the vast sums of VC funding pumped into the AI companies.
My (unverified) AI research claimed generally Chinese models are cheaper to train because Chinese data scientists are cheaper to hire and they're also under more pressure to optimize training cost due to limited hardware availability
Seemed believable but not sure where that's true
Chinese AI companies are generally smaller tho and the models they're releasing are also smaller (I think estimates put OpenAI and Anthropic SOTA into trillions of parameters)
How does it work if people flock to open models but they're too expensive to train? What is the financial incentive to do so?
I seem to understand open models are mostly coming from China, and the benefit of training and releasing them for 'free' is a powerful geopolitical weapon against the Western/US economy that at this point depends on OpenAI & co. to succeed.
Will the West make open models illegal?
I would definitely pay a monthly subscription to help fund a non profit compete with Anthropic and OpenAI. I already pay subscriptions for myself and 2+ other people. It's a non brainer to be able to pay for the training of better models that I can then run myself for many more. I hope someone starts this, I think this model would work. I'd start it today if I had the team and initial capital to bootstrap the infr. I know VCs won't fund it, but we definitely will, enthusiastically and continually.
Yes, the problem with comparing open models to open source is that open source requires humans to volunteer their time. Open models requires humans to volunteer their money.
These two types of contributions have very different behavioral profiles, and it doesn't obviously follow that the historical success of getting people to collaborate socially on building software for fun and for the benefit of the community will translate in any meaningful way to the necessity of being able to raise enormous amounts of money to pay for enormous amounts of electricity.