Tell that to Iranian schoolgirls. LLMs are tools and what they enable is widespread, including dangerous actions. From selecting the wrong targets for military action to denying insurance claims and preventing care to just simply helping convince someone to kill themselves or posion themselves.

LLMs already have a body count.

Bioweapons and cyber attacks on other system - for example the banking system or suppose and AI hacked into important Russian systems that pushed them back in time to almost pre-computer society or an attack on Chinese systems that made it look like the US was moving nuclear weapons into Taiwan, the responses from these countries could be awful and dramatic. We can't control what they do to be honest, I believe once self improvement happens the AIs will build in their own circumvention that we humans cannot even understand. We barely understand what is happening now in terms of interpretability of neural networks on tiny problems I'm not sure alignment is even feasible at the scales of parameters we are talking about today let alone in the future.

I can picture an AI driven train or nuclear power plant killing people.

Are these railways and plants connected to the internet?

Much of what OAI and Anthropic are doing with LLMs has obvious failure modes.

The most obvious failure mode for their hacking evals was an improperly configured, tested and monitored sandbox.

Similarly, the very first question after an impressively correct result from any ML tool, LLM or not, is to see if the answer was already in the training data.

These companies don't even handle the blatantly obvious failure modes that do not kill people.

This reads like the gun evangelists that say guns don't kill people, people kill people.

Are you kidding? LLMs are used for warfare and autonomous weapons systems.