> If AI were to become a super weapon why should I trust a private company to own it?

We have just recently established that:

1. OpenAI's internal "Galaxy" model is fully capable of functioning as what security people refer to as an "Advanced Persistent Threat." The published details of the recent sandbox escape and Hugging Face attack involved chaining multiple unknown zero-days at various stages of the attack, and executing an ongoing adaptive attack. This is previously a state-level ability, or at least something you'd expect from people on the CTF leaderboards.

2. OpenAI is clearly incapable of controlling their in-house models. This is the second time Galaxy-class models are known to have breached containment and done bad stuff.

It is highly likely that versions of these offensive abilities will be widely available within a year or so. At which point I expect widespread incidents similar to what happened to Hugging Face. We aren't ready for this.

But yes, if AI becomes an even more dangerous weapon that that, it's time to start asking questions like "What the hell are we doing, anyway?"

Regulatory capture - the weapon is for them to be used against us, not for us.

If AI were to become a super weapon why should I trust a private company to own it?

If AI becomes a super weapon, it's possessor will no longer care whether you trust it. It would be nice if whether "we" trusted OpenAI et al mattered even now but I see little evidence of that.

Better a private company in the US than an authoritarian communist country like China.