alt Hacker NewsThat's not what I said, and it's the least sophisticated way I saw to phrase it to engage your argument in a precise way. The closest thing to your summary that I would accept is, "words can't convince people to do arbitrarily crazy things, in a way that you can predict."
> If you had [superintelligence], you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X ... But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.
So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
Replies
>So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.
And why do you need perfect prediction and/or perfect control? "Good enough for the results you want" prediction and control sounds perfectly fine.
We have had cult leaders that had neither perfect prediction or control, and still convicted hundreds of thoysands of people to live as they say, and a decent chunk to even suicide on command!
> (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|
"You can't know what the impact of your action will be if it doesn't change much"
> (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large
"You can't necessarily change much"
> (or that f_actual(X) saturates its codomain)
"People can't do arbitrary things". I agree with this statement, but... people can build atomic bombs by themselves? See The Atomic Boy Scout. I don't think it's very relevant to the discussion that you can't make someone phase through a wall.
> when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."
"Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".
You don't need perfect prediction, you need good-enough influence on enough of the population. Which is something that adtech, social and trad media, and troll farms have perfected already.
We literally have people running around convinced the earth is flat. And a president who is both senile and delusional with access to the nuclear codes.
Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.
Or possibly inserted itself into critical elements of the tech industry. A digital adversary could do huge damage just by taking AWS offline.
That's just one possible attack vector among many others, most of which aren't well-defended. Especially if the AI itself is being used to harden their defences.