alt Hacker NewsI think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain.
With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it.
But that's just my take on this, I'm still trying to wrap my head around it all.
Yes, I think that’s a good explanation. There are really two sides to it.
There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.
The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.
More concretely, in reinforcement learning there are no ground-truth next tokens to predict.
In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.
But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.