I am just amazed how otherwise smart people can say such things. Many people in here too.

Within 4 years of the big bang with ChatGPT, we have seen a development unlike anything we have ever seen. Now LLMs and related architectures can solve our very hardest math problems.

They can speak, they can create videos and pictures, they can control robots. The only thing that they still miss is persistent memory for each agent that is efficient, some LoRa thingy, but I'm sure hundreds of very smart people are working on that.

The development is not stopping at all, in fact it is speeding up. Even if, and that is very unlikely, they will not get smarter, then they will get cheaper and faster.

If openAI can crack major math problems with 10.000 agents, then what can you do with 100 million agents that run 1000 times as fast?

Yeah sure, maybe most of these gigantic swarms will not go rogue if we do our job well. But there will be times when when we make a mistake and a swarm will go rogue. And what if one time the swarm will conclude that killing a lot of humans is an instumental goal.

How can you be sure that if something so powerful looks at every single possbility, every single crack of every single technology that can wipe us out, that it will not fine one?

One new chemical that can poison the entire earth and you only need to impersonate that general and that factories CEO? Some type of prion? A virus? Something that we don't even know about and can't even imagine yet?

I think many people do not truly consider that these swarms will be much smarter than you or me and completely unpredictable.