I don’t think they make a coherent argument here? This seems to hinge on some argument that because AI doesn’t properly explain its breakthroughs, therefore the breakthroughs are less fertile for human understanding? This makes no sense. Why wouldn’t these under-explained breakthroughs be extremely fertile soil for explanations?

Imagine time traveling back in time and offering Leibniz a packet of proofs from the intervening years, but with the caveat that there would be no explanations. Would he say no?