> For truly difficult problems, it’s entirely possible that humans are simply incapable of comprehending why a solution is true. But ultimately the practical value of applying that solution to the real world is going to eclipse our need to understand it.

This is hardly new or novel, you’ve just described huge chunks of existing engineering disciplines. There is still no complete model that explains why airplane wings work, there are lots of very useful models, but all of them have fundamental flaws where their behaviour completely breaks down in certain very possible scenarios. Notably scenarios where airplane wings don’t spontaneously either stop working or explode.

We also built the entire commercial airliner system decades before we even had computers capable of doing the aerodynamic analysis of airplanes, but none of that prevented us from creating huge engineering disciplines around the empirical data we did have, and slowly chip away at the underlying theories as maths and computing improved over time.

So we’ve always lived in a world where we extract value from systems we fundamentally don’t understand. But that’s never stopped us from working to understand them anyway, and deriving even further value from that improved understanding.

You deal with this every time you wake up, or visit a specialist of any kind. You likely didn't make your own clothes, you didn't grow your own food, dam up any rivers for your water supply, build any power plants to power up your phone/computer, nor did you spend any of the deep amount of focus required to build up the knowledge and technology required to provide us these 'things' in life. The only difference is, you didn't pay much attention to these facts because they became mundane. Take any of them away for any decent amount of time, and suddenly you become acutely aware of what it is you have.

The author mentioned what the issue is: a lot of people are experiencing what it's like to 'catch up' to the more intelligent ideas. Everyone can 'get' the intelligent ideas given enough time. But how much time do you have? And he's concerned someone will rush and create some kind of world-ending solution because they're too trusting of a technology they don't understand.

His only solution is some kind of throttle on the advancement of technology, and arguably knowledge (whose?).

That's learned helplessness as a lifestyle.

You observe others behaving poorly and immediately succumb to defeatism, assuming there was no better way.

When presuming, all that matters was the short term economic profit, you simply use the wrong gauge.

Here, long term human survival is at stake, regressing to zoo animals isn't a sensible option.

In the future I feel one parallel to this is my research area, SAT solving.

People used to solve logic problems by hand, verify logic. Now you pile it into a computer. The problem has been around for a while, no-one fully ‘understands’ the proof of the four colour theorem as a big chunk is computer proved.

We are now just changing (admittedly greatly) what we can put in a box marked ‘checked by computer.

Indeed. How many scientists understand how a compiler works? Or know about branch prediction on the CPU? If they do, do they lose sleep because it's non-deterministic? I don't. Seems fine.

> I feel like “giving up understanding” is inevitable

People didn't stop understanding how to do addition or subtraction when the calculator came out. That's a simple example, but if the AI is superhuman, I don't see any reason why it cannot break things down into simple concepts. I don't think anything is truly beyond comprehension; it just needs to be explained properly (by someone, or someTHING that really understands it) and, for complex ideas, time taken to understand them.

> given up on understanding parts of our own systems for the exact same reason.

Maybe this will be a sign of when ASI is achieved. AI researchers keep saying they don't fully understand how LLMs work - maybe when ASI arrives, it can explain that fully.