> A real-world system with the abilities of a robustly superintelligent mathematician can, when some practical problem requires it, formulate a problem statement, churn for a bit, spit out a formalized answer, and continue whatever outer-loop task it was doing without a human even finding out.

Without anyone ever finding out.

In the long run this means almost no progress past what we already have, because there's no shoulders of giants to stand on. Not even AI shoulders.