alt Hacker NewsYou are completely missing the point.
The point is that next token prediction is potentially very powerful. Nobody's claiming that LLMs can predict the market or win a chess match.
But the argument against next token prediction as a powerful tool is void.
Replies
zahlman • today at 3:18 PM
I think that you fundamentally understand "predict" differently from those making the argument.
It's the transformer architecture (token embeddings and cross-token encoding) and residual stream (cross-transformer encoding and token differentiation) that make LLMs powerful tools.
The key question is not whether this architecture is powerful. The key question is how well these models can represent things outside of their design.
With chess for example, are they actually learning the rules and strategies of the game (like AlphaZero/AlphaGo, Leela, etc.) or are they just relying on a large corpus of chess games? For example, a lot of early game moves will have been played out and the best moves will likely be the more frequent moves in the training set. So LLMs could do well early game based on that game corpus.