alt Hacker NewsThe knowledge how to build an A-bomb is also not going back in the box.
It only takes me excavating massive amounts of uranium ore, building huge facilites packed with thousands of centrifuges that span multiple square miles, and paying all that infrastructure and workforce.
Your proverbial genie can be out of the bottle all you want, but it doesn't work without getting kicked in the ass by a very large golden boot.
Bomb manufacturing doesn't scale with Moore's law.
And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)
No one knows what the limits of AI are. It's not just untested, it's unmodelled, and unplanned - build it first, worry about consequences later.