> a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games

Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:

> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]

and

> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]

The actual bitter lesson is this:

> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]

Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.

Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000

This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does.

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

It's the exact opposite.

The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.

Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.

Computer chess is the canonical example of this.

I think you have it backwards.

The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.

Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.

Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results

Chess is a brute force search problem. Humans are not good at chess, even a small computer can beat Magnus Carlsen.

It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.