alt Hacker NewsThere's a substantial gap between "the machine is better than human mathematicians" and "the machine can formalize a solution to any problem immediately." Much of the important mathematical theory behind modern science was mapped out in advance by mathematicians, and scientists were able to find and take advantage of this mathematics where applicable. Mapping out our understanding of e.g. physics and our understanding of mathematics in parallel and finding connections later has historically been a highly successful approach, and I see no reason to believe the same would not be true for advanced machines. Would Einstein have been able to formulate GR without the existing theory of differential geometry? Maybe, but maybe not. Would an advanced machine be able to develop the theory of differential geometry in response to gravitational measurements? Maybe, but maybe not.
Edit: Perhaps I could have been terser here. Really the idea is that small jumps are easier to make than big jumps. Continually developing pure mathematics is a way to make small jumps in logic and intuition add up to big jumps in applications.
I don't see a reason this dynamic couldn't continue with zero dedicated effort toward "pure math" as such. Fundamentally, the function of math in practical application is producing deterministic(or stochastic, but obviously not in the LLM sense) models that, given initial observations/conditions, are capable of predicting some aspect of the future with reasonable accuracy. In the same way that human society does, a machine would grab the physics rung before the theory rung every time, but why does that matter? If progress is stuck, the machine turns the gauge away from "exploit" and toward "explore" and put together a bag of new tools it can sequentially try on the real-world problem.
I still don't see a role for humans in this process. They might direct the practical/physical aspect(if AI turns out to be less superhuman there) but they'd likely turn the hard conceptual problems over to the machine and never look inside the box - no human alive could understand even the smallest part of what's going on in there in less than a thousand lifetimes anyway.