The odd thing is that we do train people to do real physics, somehow despite my cynical take. I see the education process as following heuristics that are believed to have some relationship to developing real world abilities, even if we don't know why. We all worked these problems, and now we're somehow able to do physics. In the end we don't know what turns people into physicists. Or musicians, artists, etc.

An analogy is making students write 5-paragraph essays. We don't really believe that the knowledge of how to write a 5-paragraph essay is a real-world ability, and the LLM's can write them all day long. But for some reason we believed that 5-paragraph essays were a good pedagogical tool.

But whether the same heuristics apply to teaching an LLM is anybody's guess. If they learn differently than we do, then feeding them on our learning tools isn't necessarily going to help them.