alt Hacker NewsIt's interesting how all the recent SOTA models are in the 1T-3T range. This even has a large number of active parameters (A52B). It is also likely that OpenAI, Anthropic, et. al. have similar sized models.
This has two effects: 1) it prevents effective use of these models on consumer-grade hardware; and 2) it pushes the requirements to train and run these models on existing and future data centres, making them more expensive to run.
It would be nice to have more slimmer models and more active research that isn't just "let's throw more compute at the problem". Maybe they'll train smaller models from the large model like Qwen have done in the past.
Replies
The hardware is certainly not cheap, but you can put 512gb of vram on your desk if you are willing to throw $30k at Apple, which is a manageable expense for a lot of small businesses. Bridging a few devices is one way to run large models, r ablation and quantization can do a lot too. You definitely need a data center if you aim to make the model directly accessible to the public, with whatever fine tuning or secret sauce you add. If you want to run it in-house as a coding engine or research tool you can likely reduce its footprint substantially.
I bet you could get the same performance with smaller models, but as english-only, by removing training data in non-english (keep code, translate in-code strings mechanically prior)
I'm not sure it really matters. I think a quantized version of a large model ends up being only slightly worse than a model trained at a smaller size to begin with. And like you point out, large models can also be used to distill smaller models, so it's good to have them out there.
This is strategic. By making it so consumer grade hardware can't serve it, you have to pay them for access to it, and that funds the server hardware market, which keeps the AI market afloat. If they made smaller models, we would all be running local AI in a few years, nobody would pay them, and they would go bankrupt (and a worldwide recession would begin). It's going to be up to us (FOSS community) to train and publish our own small models. (Also because the US and China are going to make releasing open weights illegal or highly regulated soon)
[flagged]
They are apparently planning more manageable models this month [0].
[0]: https://www.frandroid.com/culture-tech/intelligence-artifici... "It will run on a laptop": Mistral confirms a lighter Large 4