I fear the current economic and political situation of “too big to fail” the most, and I think this will be the main reason why no real safeguards will be implemented. The only reason a proper safety concept may eventually be introduced is because it will be written in blood, and I fear that by then it will already be too late.

An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.

But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.