I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.

The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.

I don't present any support for or evidence of:

- LLMs gaining intelligence by whatever means

- Extreme intelligence being possible

- Extreme intelligence being feasible

But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.

> What makes you think that persuasion isn't capped at some lower level than world-ending?

People can be persuaded to kill themselves (see koolaid). People can be persuaded that arbitrary others are evil and must be exterminated (see Hitler). People can be convinced that dying for the cause is worth it (see 9/11 or kamikaze). People can believe that bringing about the end of the world would cause them to be saved into heaven (see Christian Zionism).

Why do you think it would be hard to convince any of these people to do something really stupid?

In fact, people don't even need to want to cause destruction to do something dumb - "I'm trying to save the world with free bio-fuel, help me create this bacteria" ("and I'm very smart, smarter than Elon Musk") - boom, pandemic.

It takes only one person to fall for it.