alt Hacker News“LLMs are just X”, mostly with X = “next token predictors” is a common pattern to dismiss the power of AI with a very shallow understanding of how they really work.
It’s not wrong, but because LLMs are generators, and generation is a kind of prediction. And current mainstream models are autoregressive, which means they generate things one by one in order. But these trivia doesn’t tell us anything interesting about how they work or their limits.
It’s like saying a Boeing 777 is just a rotating machine, and it flies by just rotating some fins. Well yes, but no. With that level of simplification we’ve just ignored 150 tons of advanced engineering and physics. Similarly with token generator simplification we ignore a few trillion parameter Transformer. That transformer is more complex than a Boeing 777, and we don’t really know how it works.
A tiny ML model can do “next token prediction”. This is not as simple as that.
Replies
It's not simple to do next token prediction. That IS what is going on. You want something 'deep'. Deep things are often very not complicated.
The deep realization is that if you can predict the next token well enough, you can do things like this:
<paste the first 10 chapters of a mystery novel>. And it turned out the killer was
And if it's really good at predicting the next token, it has to understand the novel and the clues, which means understanding the context and the language and human norms and innuendo and story telling, and tropes, and red herrings, and predict who the killer was.
I think you want it to be something more complicated. It's literally not. It just turns out predicting the next token is equivalent to a universal compression algorithm, which is a form of general intelligence. And we have almost unlimited 'labeled' data to train autocomplete.
The whole point of transformers is that you can take “a tiny ML model” and just scale it up 100000x and then it tells Zuckerberg what to bake with his kid