alt Hacker News> Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?
I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.
> And the rest of us have no way to stop it or defend.
If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.
It should not be an easy choice to trust the latter. Sure, "literal terrorists" exist, but guess what? State terrorism (and state-sponsored terrorism) in the form of massacres, genocide, torture, forced disappearances, etc. is far more deadly and far more prevalent than fringe ideological groups randomly attacking civilians.
These states and elites with the access to the scary weapons ARE the ones doing the damage. The call is coming from inside the house.
I don't understand your point on nuclear proliferation concentrated in the hands of a few states preventing nuclear war or terrorism. Remember, the only two nuclear weapons used in attacks killed a quarter million people, ~90-95% civilians. By one of the few states who had the power. This is the definitional paradigm of state terrorism. And like, now look at the status quo of nuclear treaties and agreements and proliferation. Not exactly a success story.
It's obviously not worth personally formulating a "defense plan" for an AI-enabled bio attack were I personally to have access to SOA weights, but if history is any indication, I feel pretty confident that the likelihood of that happening remains far greater in the closed-weight, elite-state-access-only scenario than the open, democratized, and collectivized one.