alt Hacker News> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue
I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move on. Only to repeat the same cycle without considering that there may be something far more serious at play.
These are AI security experts, and this has been their way of "solving" these incidents. AI security experts ...
Moreover, when Challenger exploded, the government launched a series of investigations into the incident, bringing in experts from across the field. And now, what has the government done? Nothing. Literally nothing, as if everything was fine and all under control.
Seriously, I'm generally quite optimistic and I don't buy into this fatalistic narrative about our shared future. But I have to admit that sometimes I feel like I'm stranded on a planet of primates.
Sorry for this rather unproductive rant.
Replies
> I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them.
Humans tend to be terrible at being proactive, but we respond fast during disasters. I think Eric Schmidt's prediction is most likely: we won't take reasonable action to control access until there is some kind of disaster. We hope it's small enough to not kill too many people, but large enough to cause widespread panic. We hope that it happens soon, because if it happens two years from now, it's likely too late. AI will be so advanced that we have no hope of understanding its motivations. All reasoning will be completely opaque to us. It will be building newer and better versions of itself using moral frameworks it itself decides. We will be completely out of the loop, and potentially superfluous to its goals.
It does seem bizarre that major AI development (and autonomous robot development) hasn't been nationalised yet and treaties drafted up around producing it. Commercial incentives just seem totally at odds with the public good when it comes to controlling and regulating something like this.
Even if not for the sake of avoiding a mitigable disaster, there's a huge benefit in simply pacing development so as to not completely freak out society as they stare down the barrel of mass job market changes without time to adapt or prepare. Nobody wants to live in a world where they might wake up in a month and find their entire industry has been automated overnight.
We've tackled much bigger global issues successfully in the past and the US still has enough pull that it could probably get most western nations to march in line. It seems like the biggest issue is belief in the right of governments to govern.
The US government is run by idiot with dementia. No wonder govt does not take any real action.
Internet isn't real.
People (well, public discourse) have got extremely bad at dealing with forseeable risks and their mitigation. You can see this in things like climate change and vaccination, but also in discussions around regular crime, food poisoning, industrial accidents, and so on.
Nothing will improve until something explodes on live TV. And it has to be something important, which means it has to be in California or New York.
They don't really care. It's just a play to tell people what they want to hear while they chase the money.
Well you have people in this very thread calling all of this a nothingburger.
Ultimately these are unserious companies ran by unserious people. They don’t even have a business plan, why would they bother with some kind of sensible security policy?