alt Hacker News>Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.
That's the danger with creating a super intelligence, that you won't understand it. My dog is pretty smart, has good instincts and I'm quite confident he can survive in nature all by himself. But if a human decided to harm him, he doesn't stand a chance.
Replies
kragen • today at 5:02 AM
To be fair, he could probably fuck the human up pretty good if he had a mind to, too. Which may be the most likely reason that a human might decide to harm him: if he bites a couple of children, for example, or if a burglar wants to rob your house.
> That's the danger with creating a super intelligence, that you won't understand it
This argument is sophistry because it assumes the conclusion in its premise. It works backwards from extinction by proposing a hypothetical computer program that definitionally cannot be understood, predicted or controlled.
A substantive argument would propose reasoning that explains why we should believe such a program could be created.