What makes you think that persuasion isn't capped at some lower level than world-ending? There's a lot of incentive for ads to be as persuasive as possible, but they are so far from ending the world that it'd be science fiction for an ad to be that effective.

If you're just talking about convincing a few individuals to do experiments, then there are plenty of people who fall for scams everyday.

> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.

Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.

>persuasion

Is there evidence that such soft skills are improving over time? I take as evidence to the contrary the fact that LLM writing has not substantially improved. I don't see that its powers of persuasion has either, whether that's through making strong arguments for a course of action or writing persuasive essays.

IMHO, one of the top-p vectors is probably something similar to what we've already witnessed with HuggingFace, (e.g. as part of an eval or RL run for cyber; things like Daybreak and Mythos are product lines), just worse in scope and somehow eluding monitoring again.

Not even that much persuasion is necessary.

We have the medium of persuasion: it's called money, and people do ALL SORTS of horrible things that they wouldn't otherwise do when incentivized by money.

We also already have the mechanism of a machine incentivizing and directing people with money, and that's called a corporation (the original paperclip maximizers, by the way).

So you just need an LLM CEO of a legal corporation with financial resources -> near unlimited possibility of real-world impacts over time.

> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.

That feels like a bait and switch slyly introducing a whole new dimension. The discussion started with establishing that LLMs are extremely helpful with AI research. What is the evidence to justify rapidly improving persuasion skills? Even the scenario you've outlined hinges on a human being stupid enough to take LLM advice wholesale -- and every conversation that hypes up AI takes us one step closer to a world where some susceptible human feels amenable to that.

> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion. A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.

Is there any evidence, either by way of quantifiable data or by step-by-step thought experiment, that suggests this is possible?

I consider it supremely silly, for example, that there is a graph of destructiveness as a function of persuasiveness where "y = manifest human extinction" is reached at some finite level of persuasiveness, as though persuasion is a single scalar that grows without bound, and the only safeguard between a conversation and WWIII is just a mind waiting for the right argument as expressed in a ChatGPT transcript. (Keep in mind, also, that you have already ceded to the OP that autonomous intellectual and physical self-improvement at singular trajectories is implausible.)

LLMs aren't magic, there's nothing they can do to persuade people to do weapons testing, it's an absurd scenario.

It would be a human guiding the ai to guide other humans to end the world.