alt Hacker NewsI think points that deserve more attention in the current public discourse are:
- This should be a huge wakeup call for everybody.
- We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something.
- It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox and roaming their internal network?
- What if in the future similarly misaligned AI agent tries to export its own weights and hack and clone itself into instances at various cloud hosting providers? Suddenly we might be dealing with a persistent threat harder to contain.
- The OpenAI post about this shows surprising lack of ability to see the seriousness of all this.
- For their models this isn't just an unlucky incident: it seems there have been multiple such cases recently, e.g. https://openai.com/index/safety-alignment-long-horizon-model...
- The fact that it happened again seems to show their lack of ability to derive useful oversight measures.
- Or they just don't care enough?
Replies
Very much agreed on the significance.
The lack of a true airgap should have been identified as a critical weakness and addressed with not only additional layers trying to prevent escape, but at minimum an alarm which would page a human when escape did occur.
My guess is this occurred in a setting where, to be frank, there were too many researchers and not enough software engineers and SREs.
All of the systems which were initially built largely or exclusively by researchers - inference, evaluation, training - are at the level of complexity and significance that they need systems experts. Maybe some teams don’t have access to them. I know plenty of software engineers are employed at OAI but I’d wager they’re concentrated in inference and training, rather than evaluation?
The ironic part is if you had presented this setup to chatGPT and asked how to improve it and if it was good enough, you’d have gotten a ton of actionable suggestions which would have mitigated or prevented this.
> we are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something
This is just laughable.
It's a fake PR issue. It's hardly the first time this happens, but of course OpenAI, with its IPO now more in doubt than ever, had to claim this (and, once again, I have trouble believing Sam Altman choosing this: this could lead to OpenAI getting regulated, which has at least as much potential to lower their IPO price as to raise it).
But there have been messages about LLMs, especially coding agents, "grabbing root" etc many times. I have experienced such an oops. Such a hack has happened and been reported on this very site:
Move fast and break things! Chances are that nobody will ever be held accountable and when push comes to shove, the tax payer will bail you out.
> The fact that it happened again seems to show their lack of ability to derive useful oversight measures.
I think OpenAI likes the attention and did not try particularly hard to constrain the setup, even when it went off the rails. Also, the whole point is to see how good the models are at exploiting stuff when unconstrained. Turns out: quite good, as expected.
Let me restate what I said in the other thread: Would this have happened if the instructions explicitly said to stay within the sandbox and that all of the (ExploitGym) solutions would be invalid if the system used information or tools from outside the sandbox?
It seems fairly probable that such instructions were not in place.