Bomb manufacturing doesn't scale with Moore's law.

And everyone had a fairly good idea what fission and fusion bombs would do once built. (Teller was worried Trinity might set off a nitrogen fusion reaction and kill all life on Earth, but Bethe and others proved him wrong before testing.)

No one knows what the limits of AI are. It's not just untested, it's unmodelled, and unplanned - build it first, worry about consequences later.