I think we have very good reason to fear big accidents. But some AI algo going rogue isn't exercising real societal power. Power-seeking and self-preservation is nothing special. We all do that. That doesn't mean every group of intelligent people with a bit of money is always on the cusp of world domination.
Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.
Agree. We already have a planet with eight billion human level intelligent beings on it. Despite some close calls in the past, none of them have yet managed to wipe out humanity.
Every random group of intelligent humans who is motivated to seek world domination is constantly in a fight with all the other groups of intelligent humans who are also motivated to seek world domination. They fight as peers.
Despite this, a minor technological and social advantage allowed one small sheep-filled rock in the north-east corner of the Atlantic to hold 25% of world GDP, population, and land area (though not all at the same time), and to hold on to >20% of world GDP for about 70 years.
For better and for worse AI, as it currently exists, is not shaped like our minds. Better for us, because machine learning needs a lot of examples, so if we invent a different behaviour or create a different environment, a machine would struggle to cope with it, this is also why they needed to read basically the entire public internet to get good; worse for us, because the hardware it runs on comes with an absurd speed advantage, which is why they are able to read the entire public internet and by extension when they got good they got good at basically every subject at the same time.
And:
> Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.
Great, but so what? We have lawyers doing this, too. Trump's tariffs on the island occupied solely by penguins looked suspiciously like it came out of an LLM (even despite them being worse then than now).
And LLMs are demonstrably more persuasive than human writers, so it's not even particularly unbelievable at this point (even if it ought to be, c.f. chatbot psychosis) that an LLM will simply convince people to give it the right to form a government and wield such powers.