Some of the most reassuring and scariest things you can read are about the incidents that have already occurred where computers said "launch all the nukes" and the humans refused. On the one hand, good news! We have prior art that says humans don't just launch all the nukes just because the computers or procedures say to. Bad news, it's been skin-of-our-teeth multiple times already.

https://www.warhistoryonline.com/cold-war/refused-to-launch-... - This isn't even the incident I was searching for to reference! This one was news to me.

https://en.wikipedia.org/wiki/Stanislav_Petrov#Incident - This is the one I was looking for.

I am scared of two things.

First, people being rubber stamps for AI recommendations. And yes, it is not unreasonable that in a dire situation, someone will outsource their judgment (day).

Second, someone at the Pentagon connecting the red button to OpenClaw. "You are right, firing nukes was my mistake. Would you like to learn more facts about nukes before you evaporate?"

If you think humans are going to delegate reasoning and responsibility to something, shouldn’t you also be concerned about the sorts of recommendations that thing is going to make?

One can try themself, for Claude is fine at waging war [1]. Notice the thoughtful UX, including the typing "I ACCEPT FULL RESPONSIBILITY".

[1]: https://nitter.poast.org/elder_plinius/status/20264475874910...

The speed with which my technical cow-orkers and friends have started relying on the "AI Overview" only, in lieu of following any links, in search engine results (let alone not using search engines at all over chatbots) tells me reasoning and responsibility will be outsourced as soon as possible.

Humans are fundamentally lazy. The brain is an "expensive" organ to use.

Trump's Golden Dome is literally advertised to help the U.S. win a nuclear war by leveraging AI.