I think we have very good reason to fear big accidents. But some AI algo going rogue isn't exercising real societal power. Power-seeking and self-preservation is nothing special. We all do that. That doesn't mean every group of intelligent people with a bit of money is always on the cusp of world domination.

Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.

I think there are a couple of interesting scenarios where humans essentially “hand over the reins” to AI.

1. A leader rises to power who has instructed an AI “I want to control the world, tell me what to do”, and has some kind of real-time capable reasoning engine in their ear/head. Even this would require monumental advances in physical compute and HMI, but I wouldn’t rule this out within 20 years. However, this still requires a human to make the initial prompt and carry things out. And in such a scenario, there would be others with the same technology to counterbalance. I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.

2. An entire society hands over the reins of governance to AI. Basically some country decides that AI has advanced to the point that it makes better decisions than any human / group of humans, and allows it to govern their country with an objective of “take over the world”. This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.

If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.