alt Hacker NewsAbsolutely it will. You have to run them in an environment without tools because today they just turn on stockfish if you ask on claude.ai.
These models are strong at math and coding because they have been heavily trained for it in post-training. That is where a majority of the gains have come from in the past two years. That has absolutely no impact on chess ability. Heavily train a deep learning model for chess and you get Leela, an absolutely monster. Don't train them for chess and I doubt its even 1200.
It is possible to do post-training on a base LLM and get a strength equivalent to a strong amateur but people haven't done that at any meaningful scale recently, so hard to say what the ceiling might be with the largest models. Probably pretty high but it would be expensive and still worse than Leela.
The only interesting question is could it explain chess to you - and I haven't seen evidence that it can. ChessGPT at least - while purportedly 1800 strength, can't explain strategic themes from the first chapter of beginner strategy books accurately or explain why specific moves are bad (positionally), it just knows they are.
Yep, they are quite bad without stockfish. You can test it with the playchess skill in my repo. Maybe 1200/1300, who knows? Still I do think that they can, with enough time, explore multiple variations where they confront their naivety to stockfish and build up a compact picture on why move Y should have been played instead of move X. That was my intuition when building this skill.