alt Hacker News> More importantly, what's the point of building monster-scale data centers on unprecedented amounts of debt when a more than good enough model runs on a GPU from a couple years ago?
Probably because the future "monster" models will be insane. 100T+ param models might be the type of things that can independently run a small business, which means anyone not using them is at a distinct disadvantage to their competitors.
The top model from 2025 looks silly compared to the top model of the first half of 2026. Do you feel like progress has stalled?
> The top model from 2025 looks silly compared to the top model of the first half of 2026. Do you feel like progress has stalled?
I do. Pre-training is where the industry saw the “emergent properties” of LLMs arise and, for a time, people thought you could just keep scaling up bigger and bigger models but then the incremental gains from doing this did plateau. Labs will still do bigger models (Bytedance has a 10T planned), but these are sparse architectures and they aren’t going to have mind-blowingly greater intelligence. Fable didn’t either.
Scaling pre-training tokens also flattened out. What is still delivering gains is scaling RL on verifiable tasks. But that’s not general intelligence - it’s fitting models to specific tasks, which ML has always been good at. More importantly, most tasks to which humans apply their intelligence don't have computationally verifiable answers.