logoalt Hacker News

bloafyesterday at 11:45 PM1 replyview on HN

I think the central premise looks more like this:

Biology has been trying to grey-goo the world for billions of years, but it turns out the world is not something so trivial.

He is pushing back against the folks with pure CS backgrounds who think that computers are all there is. Its a form of magical thinking unique to programmers who live in a world where speaking the right words to a machine is enough to impart your will on the world. Believe that strongly enough, and you fall into the trap of thinking that a sufficiently smart entity could speak the words "let there be light" and it would be so.

The author is pointing out that speaking the words is insufficient. To end humanity there must be an execution phase. The author is correct to point out that acquiring superhuman intelligence is not some guarantee that you will have or obtain the resources necessary to make that happen, in the same way that genius generals still lose to ordinary ones, and the best-laid plans are oft to go awry.

There certainly are risks, but 10% risk of extinction in 10 years is not one of them.


Replies

dumberquestionsyesterday at 11:58 PM

Biology is extremely slow, try to extrapolate the previous 3 years of AI progress 10 years into the future, it's not a comparable trajectory.

show 1 reply