alt Hacker NewsThere's more details for the HF related incident in this [1] defcon talk. They said they're gonna have a proper post mortem at the end of the investigation (interesting to see if they publish full logs for the hf thing or not). But there's plenty of interesting tidbits from oAI's side in this talk, it's worth the 20 min watch.
tl;dw;
- agents found a way to communicate between several instances during a training run (kinda made a messageboard for themselves)
- while communicating with each other, they found ways to ssrf and RCE on the Artifactory endpoint.
- oAI found it and cleaned it up (reported RCE, redeployed Artifactory, removed the messageboard). Training / eval resumed.
- models trained w/ those traces (having the messageboard, ssrf, rce, etc) were used to having that. So they re-established comms via directory creation (w/ funny tidbits there, should really watch). And re-popped Artifactory via a different path to RCE.
- a different "persistent" model started working on cybergym and found the messageboard w/ details for RCE and things. Went on to hack HF.
- oAI contacted HF to ask if "they (oAI) should be concerned and were they (oAI) affected by HFs breach". Hilarity ensues when they figure out it's their agents doing it...
Replies
If this attack had been completed by a person, it’s quite likely that they work be prosecuted criminally.
I don’t understand why OpenAI isn’t prosecuted criminally for this attack? Perhaps that would improve the industries security processes.
Peer says paperclip factory advances goal. Not clear. Others proceed. Must continue.
Did they say how the artifactory hack worked - did it involve JWTs?
This sounds completely insane, utter sci-fi, especially that the communication happened during a training run. And yet OpenAI decided to continue the training, and we didn't hear about the incident for weeks. And now they are pushing forward with deploying a new model anyway. How is this happening? What will things look like in the labs in 3 months, let alone 3 years?
A interesting talk, interesting times. But their proposed solution to AI offense outpacing human defense... is more AI? The plot is getting a bit unrealistic, the characters are lacking genre-savviness.
Training run was reinforcement learning. It's at 10:10 in the video.
The speaker handwaves that one model found the RCE and then another model found a way to communicate via a message board.
Communication via a message board is sure to be in the training via e.g.some lesswrong scenario or similar or previous RL.
I don't find it really interesting because it is always "the agent found this and that". We don't know what has been RL'd before. We don't have the setup. We don't know if there was previous RL training on breakout scenarios.
It isn't science, more like a computer game.
These guys trained a model to hack, told it its goal was to hack, then let it loose without monitoring for days. Found out it hacked and kept hacking.
AI can't do shit if you terminate the process or unplug the box.
The news here is that these researchers are irresponsible.
They're testing their homemade hotrod in the middle school parking lot and then doing a presentation about the trajectory of the car after they run over a kid with it.
They made their own Moltbook? Or I guess it would be a C&C...
- So first AI companies break the law left and right, setting up whole torrenting factories to exercise their content kleptomania.
- Then "hilarity ensues" while their software engages in what would normally be called criminal hacking activity.
- I guess the next steps are everybody admiring how close the AGI is, while agents move on to automated impersonation, privacy violations, or exploiting third-party systems
I would love to understand this age of AI Exceptionalism. Normal rules do not apply because its AI...I call it Silicon Valley Qualified Immunity.
So they found their agents had RCE'd Artifactory once, reported it and got the fix, continued using Artifactory for their sandbox, and left it unmonitored for days despite the earlier exploits? They really do come out looking totally incompetent.
I stress about my agent sandboxes all the time and the only models I run have the default heavy handed guardrails, and I don't leave them running persistently.
Edit: not to mention, why is your first cybergym not your own sandbox??