But to extender your analogy: We do understand how Claude does what it does and how it is built, so how is this different?
We definitely don't understand how Claude does what it does! We know how to grow a Claude, but we don't understand what goes on inside a Claude.
I remember when about 15 years ago, I wanted to make my neural network home work based on proven math. I had to train a model to learn basic number recognition. The math said 100000 neuron is what's needed, so I typed in that, and my laptop immediately froze completely, and died a little later. I learned on that day that we know empirically that a few 10s are enough.
We have the same thing with LLMs. We don't know how they work. We don't understand them. We have only empirical, statistical indications, nothing is proven. We know how the small parts behave, we really understand those, and also the different algorithms. But they whys? No. We're getting better, but still far away.
And presumably mathematicians understand the output, the proof, whatever it is that came out of the process. I don't feel like humans have to be the one doing all the manual math and figuring out all combinations.
I mean I understand how cars work but that doesn't mean I'd feel the same accomplishment of driving 42 km as running a marathon.
I would dispute that we completely understand how LLMs do what they do. An analogy would be that we meet an alien race that gives us their proof of the Riemann hypothesis translated to Lean. Plus of all other provable mathematical conjectures we come up with. And we might kind-of understand how those aliens do math. It would still be a bit sad that we'd be relegated to consuming what the aliens give us, and never be able to ascend to the level of the alien mathematicians