logoalt Hacker News

mrobtoday at 6:45 AM1 replyview on HN

The way I see it, there are three possible outcomes:

1. AI works worse than expected.

Our economies are depending on this not to be the case, so it triggers the Greater Depression. Widespread poverty and misery ensue.

2. AI works as exactly as expected.

This means whoever controls it gains enormous power over everybody else. There's no possibility of resistance: the Second Amendment doesn't matter when your oppressor has fully automated murder drone factories. We enter a dystopia beyond anything Orwell imagined. Note that this is an arms race, which means there's no limit to resources it can consume. Billionaires are fighting over who gets to be king of the world and they don't care how much you're paying for RAM.

3. AI works better than expected.

This means the "recursive self improvement" plans succeeds, and the "intelligence explosion" scenario happens. This, with probability very close to one, results in the sudden extinction of all life. Human values are a highly complex result of our shared evolutionary history. Something that did not share any part of that history will have profoundly alien values, e.g. "minimize training loss". If it's vastly more intelligent than us, it will be able to fulfill those alien values which extreme efficacy. There are very few goals of the "make number go up" kind that don't result in everybody dead when taken to the logical conclusion.


Replies

queenkjuultoday at 10:06 AM

But our oppressors do have automated murder drone factories and have for years. AI doesn't need to work for billionaires to keep building their dystopia.