These AI bros don't get it. These LLM systems solving math problems have the benefit of being constrained by formal rules of math providing constant feedback when they break the rules that prevents them from wandering into la-la land. As Dijkstra said in his prescient "On the Foolishness of 'Natural Language Programming'":
> "The virtue of formal texts is that their manipulations, in order to be legitimate, need to satisfy only a few simple rules; they are, when you come to think of it, an amazingly effective tool for ruling out all sorts of nonsense that, when we use our native tongues, are almost impossible to avoid."
To whatever extent that LLMs can "think, reason, or be creative" in generating these proofs (or writing code), they're aided by very high guardrails so it doesn't say very much about what they can do in broader, fuzzier domains.
Would you say that the early results of AI beating humans at backgammon and checkers had any predictive power towards chess and go?
Goedel proofed that you can formulate nonsensical statements in math (and any formal system that is complex enough to be useful)