alt Hacker NewsThe discourse around this is annoying. A big group of people use "next-token predictor" to imply that LLMs aren't capable of anything interesting. Another big group of people opposes the use of "next-token predictor" because of that implication. But that fight isn't about the "predictor" language at all.
The linked article makes a good point: a substantial chunk of the training does not consist of "here's a bunch of tokens, here's the next token, learn that." But all the comments want to turn it into a referendum on the goodness of AI.
Replies
You can pour a bunch of beads into a pachinko board and end up with a normal distribution. Do the system “know” they need to go there?
Then you feed a bunch of tokens into a GPU and end up with a distribution of possible next tokens…
The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things are made. In pre-training the model really is trained to predict the next token. What is being emitted by the model is, by structure, by training and by optimization, a prediction of the very next token.
What is emitted by a model during RLHF and RLVR is not, by structure, training or optimization, a prediction of the next token.