Yes I understand the analogy was a bit loose. I'm comparing what happens at "inference time" in chess engines to what happens at train time in LLMs. In hindsight AlphaGo Zero was the perfect analogy, but I missed that opportunity.

The analogy with chess still works, but there's an extra step to think about. In both cases there is some kind of search over possible future trajectories. A chess engine explicitly searches branches of the game tree and evaluates which moves lead to good outcomes. In RL for an LLM, you sample rollouts, evaluate the resulting trajectories, and use those evaluations to update the policy.

The extra step with the LLM is that you don't keep doing that whole search at inference time. You use the rollouts to update the weights, so in some sense the useful information from that search gets compressed into the model.

But if you accept that the model is, in some loose sense, storing what it learned from those rollouts in its weights, then at inference time they are doing a similar job: taking some input state (prior tokens or a board position) and choosing the next action.