Should we unban the development of nuclear weapons then? Somehow I don't think that humanity's strength was adaptation in that regard. We banned a dangerous technology and were better off for it.
We didn't ban nuclear weapons, we capitalized on them being extremely difficult to build.
AI doesn't afford us that same luxury of nuclear weapon deterrence.
Bans do don't shit to stop interest parties.
I don't think nuclear weapons were banned out of moral enlightenment... They just have 0 ROI for the private sector, there is no market for them. And they are bulky, difficult to make, and require a bunch of stuff like centrifuges, it's a much simpler task to agree upon. If you instantly got 2bn$ for every detonated nuke, I think the world would look different.
It's different with code and LLMs because there is actually a market for this. It's more like you are trying to ban excel spreadsheets or something. They are already out, a shitload of people use them, and you get outcompeted by countries/companies that don't ban them.
1960: ~20,000 nuclear warheads around the world
1967~1970: The Treaty on the Non-Proliferation of Nuclear Weapons was signed into force
1980: ~54,000 nuclear warheads around the world
That was how we "banned" nuclear weapons. When you hear "ban A" you should automatically translate it to "ensure that only a few most powerful entities can have A" in your head.