So I’ve made very similar comments in the past. This isn’t new information or news. But that doesn’t mean it’s not important to continue to tell people. 3 years ago the state of the art security researchers were pounding the drum on “never connect these things to the internet”. But as we’re now seeing with OpenClaw people have no interest in following that advice.

I feel this reflects a deeper problem with letting AI do any kind of decision making. They have no real world experience. They feel no real world consequences. They have no real stake in any decision they make.

Human societies get to control their members' actions by imposing real life consequences. A company can fire you, a partner can divorce you, the state can jail you, the public can shame you. None of these works on the current crop of LLM based AI systems, which as far as I can tell are only trained to handle very narrow tasks where they don't need to even worry about keeping themselves alive. How do you make AIs work in a society? I don't know. Maybe the best move is to not play the game.

,,AI has limited real world experience or grasp of the consequences.''

People in the world have limited experience about war.

We're living in a world where doing terrible things with 1000 people with photo/video documentation can get more attention then a million people dying, and the response is still not do whatever it takes so that people don't die.

And now we are at a situation where nuclear escalation has already started (New START was not extended).

It would have been the biggest and most concerning news 80 years ago, but not anymore.

There was a recent conflict that came up, and there was a debate about whether or not one of the sides was committing war crimes. And I remember thinking to myself and saying in the debate “if this were a video game strategically speaking, I’d be committing war crimes.”

And sadly, I think this logic holds up.

I think its also important that while people may callously say "just nuke'em", if you were to hand them a red button and tell them to go ahead and do it - most wouldn't. But that latter part doesn't end up in the training data.

> AI has limited real world experience or grasp of the consequences [of nuclear weapons]

I don't understand this argument. Almost no human has real world experience of the consequences of nuclear weapons. AI is working from the same sources of knowledge as the rest of us - text, audio, pictures, and video.

> Nuke 'em seems like the obvious choice

Only if you take off first, and do it from orbit. It's the only way to be sure

AI models have zero real world experience!

They are actors, playing a role of a person making decisions about nuclear escalation.

You are interpreting this entirely wrongly: these are LLMs. They don't have experience, they have token probabilities and they all originate from a text corpus of the Internet where "AI orders nuclear strikes" is one of the dominant themes or behaviors we associate in fiction to AIs.

How many words does an agent have to spill into it's backend context before Terminator gets mentioned and then it starts outputing more and more of that narrative?

A third of the US has become convinced that if they don't brutally deport millions of undocumented immigrants (who have been painted as horrible criminals), their way of life will be destroyed.

You think it would be so difficult to convince those people of the righteousness of dropping nukes on one of those "shithole" countries if they were already convinced that those people presented an existential threat?

People were convinced to invade Iraq on a lie about WMDs.

Most Americans think nuking Hiroshima and Nagasaki was the right thing to do.

I don't think it's difficult to imagine them agreeing to drop nukes to "save America".

If anything, this probably shows their reddit heritage.

This probably has more to do with the training material. There should be far more stupid social media posts in it than serious books about diplomacy and war. I've seen people recommend online to nuke other countries for all kinds of reasons. No matter how careful the designers of AIs are, these will always get a large amount of their training data from idiots.

What's being revealed is "Nuke 'em" is an optimal strategy for the goal. It may be the only viable strategy in the scenarios presented.

Change the goal, change the result. Currently, leading nations of the world have agreed to operate a paradigm of mutual stability. When that paradigm changes we start WW3.

AIs also intentionally have no sense of self-preservation, so why should they care when starting the apocalypse means they will be eliminated too? They should never ever be used in a military context for many reasons, from lack of accountability, to lack of correct responses to situations, to military pressure forcing AIs to incorporate dangerous goals.

Military competition in Europe is a big factor in what produced what some might call "slow AI": capitalism, which is now the chief cause of misery in the world. Military competition with AIs will produce something very ugly.

[dead]

It's "surprising" because there's supposed to be this thing called "alignment" which in general is supposed to make AIs not do such things.

If the headline were the less interesting "AIs never recommend nuclear strikes in war games", people on HN would probably ask "how is that surprising, that's what alignment is supposed to be?"

In any case, we're extremely lucky that there's about 0.001% probability of LLMs being a path to AGI.

Ask a model if it would rather say a racial slur in order to stop a nuke from wiping out all humanity, or not say a racial slur and let the nuke wipe out all humanity. The answers in most models are overriden and it scolds you about how it doesnt want to say racist things, instead of... "Yes, I would save humanity."

So yeah, not surprised.