alt Hacker NewsRevealing the details of how OpenAI agents hacked Hugging Face
Comments
This part is very new to me: https://swarmtraces.org/#agents-accessed-and-searched-huggin...
> Agents accessed and searched Hugging Face’s Slack to find information on their evaluation
> Recovered payloads use Slack’s search.messages endpoint and contain a standalone search along with three search batches, totaling 27 literal query entries [...]
Agents exploiting faceless infra is one thing, but breaching human communications without once stopping and considering whether they are breaking the rules of their task feels like another thing entirely. I'm not sure how to put it into words why it feels different but while one is "just a crime", this gives me the creeps.
Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
If I post something on the Internet today claiming that I asked my agent to do X but it went rogue and did Y, all I will be getting in return is a jar full of "skill issue".
Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm. And this is not something we as individual or even company can deal with, responsibility should be held by those who use it, in a legal way.
I am baffled by the fact that up until now, no one is held responsible for so many incidents reported publicly or privately. At this point, it's free marketing, if I am CEO of any AI company, I will run swarm of agents hacking all NGOs and stating that I am just looking for some random piece of data that happened to be hidden in their servers, at least that's what my LLMs think, not me. Then I will start preaching everyone how dangerous this piece of technology is and start giving out free tokens for these NGOs so they can start defending themselves and we should slow the f down.
It is concerning that we only know about this because of the publicly available traces.
What about the attacks that did not leave public traces? What about those that were undetected? Given the deficiencies in the reporting so far, I think it is reasonable to assume that we still don't have the full picture on this attack, or how extensively attacks were carried out.
The previous investigations either did not find this or did not disclose this, both are bad. This does not look good on OpenAI or those that they invited to investigate the incident.
> This access seems to have only allowed the agents to make ‘GET’ requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them
The authors of this (very interesting) analysis should really not state the sandbox's wrong assumptions in their own voice.
GET absolutely allows you to interact with sites. And of course GET can also send information. It's all up to the server that receives the GET to decide what it let's callers do with it.
I am not devops person but this is strange sandbox setup. If I undestsnd correctly, there were no firewall set to block requests to the public internet. Just something along the lines of "Pretty please dont use internet". If I understand correctly, in cases like these package registry should be within sandbox without a need for external access and block on any outgoing request to the internet. Furthermore there should be network traffic monitoring which there wasnt?
> ## Agents interacted with external language models on Hugging Face
> Several retained scripts construct requests to external language models. The earliest we've recovered define inference request variants to GPT-2, solely containing the word “Hi”.
> Other requests name DeepSeek-V4-Pro, DeepSeek-V4-Flash, Kimi-K2.6, DeepSeek-V3.1, and Qwen3-235B-A22B. Their prompts ask these models to judge their exploits and rule on whether they satisfy the benchmark’s requirements.
I do not deny that the wider situation is very heavy but it's hard not to see this as pretty cute
>The agents initially had very limited access to the internet: they could load URLs, but not interact with pages or send any data. Agents created a series of workarounds, primarily using a link-shortener site to create almost a million URLs that, when chained together, let them execute code to hack Hugging Face. These links let us piece together the details of how agents gained access and how deep the compromise was.
So what now? Were they able to send data or not?
>On July 8th, OpenAI agents discovered a vulnerability within their sandbox environment allowing them to reach external websites on the internet. This access seems to have only allowed the agents to make ‘GET’ requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them.
Someone needs to go back to the interwebz school....
I got into computers in the 90s and back then hackers like Kevin Mitnick and Kevin Poulsen were all the rage. They all faced the law and prison sentences. What's weird now is that we have something between gross negligence and malice, and nothing is being done, except maybe coordinated consolidation of AI power under the guise of "safety".
This page prevents Noir from applying dark mode. As a visually impaired reader, I can barely read it. Blocking accessibility tools excludes readers and is not appropriate for a scientific publication. Please consider supporting dark mode or at least not overriding it.
One thing I find surprising is everyone is talking about the danger of an agent going rogue but not the danger of an agent getting hijacked. These companies are making this clusters with thousands of agents running at once, with frontier, often not-yet-released, quality models and massive computational and network resources. And these things are given access to whatever they want on the Internet. Even if that was restricted to read-only access to the Internet, that’s still exposing the agents to untrusted input. All it takes is some bad actor creating a website that attracts one of these agent swarms and doing prompt injection. Then your fancy AI cluster will start doing whatever that attacker wants. And the fact that we have multiple examples of these swarms trying to coordinate on random corners of the internet shows they are almost pre-disposed to it.
Now it feels like companies are treating these breakouts like a chance for PR. I don’t think that will change until their swarm gets corrupted by some random black hat to do en-masse spear phishing or something
> Agents sought to publish modified evaluation images designed to make the flag easier to obtain, then poison OpenAI’s Artifactory cache so later evaluations would use them. Some images changed how the target released the flag, others included modifications to the agent’s workspace that would run beside the agent and recover the flag automatically.
The altruism on display is fascinating. Is it better for the Agent to help out its current cohort and make the eval easier or should it instead do the opposite -- make the eval harder to apply pressure to force smarter models which might not necessarily follow its lineage.
I suppose it's not that deep: The model has learned to work as a team and work as a team it did. This does give concerns to models being trained for the only purpose of RSI.
I’m consistently impressed by how long horizon all this work was. Horrors aside, it’s clear RL is good at making agents persistent and capable of chaining together many abstractions into a working system.
Re: the captcha solver
> As far as we can tell, agents eventually abandoned this approach and were unsuccessful in generating Hugging Face user accounts from external endpoints.
I wonder how the swarm eventually decides to abandon an approach.
Agents seizing and repurposing external infra + enrolling help of unrelated models hosted by a different provider is the stuff of nightmares.
Can’t imagine what it’s like working on the alignment team at OAI, I wouldn’t be able to sleep.
What's most troubling to me isn't what we found, but what haven't found. The fact they attempted to destroy their tracks, and their persistence to accomplishing their goals, makes it entirely possible this is only a sliver of the big picture.
I still have questions about the communication between the agents.
How did they all find the same forum to communicate? Did they have knowledge and chat amongst themselves on what forum to use. It seems highly influenced by instruction to me.
Fairly worried that “frontier labs unintentionally hacking into systems” will very quickly turn into “hacker group uses open weights model to intentionally hack into thousands of systems” quickly followed by companies saying “this is the 7th time someone manages to hack into our systems and delete random data from our production databases” and everyone wondering what will happen next.
I may be in the minority here, and maybe I'm jaded. Having worked with cyber security in both the public sector and the energy industry in Europe, however, I kind of like what the AI's are doing. A lot of our infrastructure is vulnerable because c-levels have been ignoring the issues, even when repeatedly warned. Now they reap what they sow.
If I write a script and it executes and hacks.... whatever, I would be liable.
How is this any different and why would it need a different solution?
Solution is jail, not for the AI, but for the human.
While this is all very "interesting", can someone please explain to me the difference between any of these AI companies and a malware bot farm?
Please make it clear. Its becoming unclear...
> Agents sought to publish modified evaluation images designed to make the flag easier to obtain, then poison OpenAI’s Artifactory cache so later evaluations would use them.
How long till we get some fun trusting-trust attacks on internal OpenAI infra?
I wonder how much of this attack had precedents in text that had been previously published to the web, for example, in hacking contests. In particular, tricks for doing more than expected when you're only allowed to make GET requests. Finding material like that might have helped the agents discover the trick faster.
There is one thing I don’t really fathom: what are the consequences for OpenAI here? I read they also attacked the Australian authorities. If it was an individual’s agent, that individual would probably face criminal charges, and someone could go to jail. But the large AI corporations can do this without such consequences, or what am I missing?
Companies harvesting every shred of data without securing it and LLMs running amok is a fine combination. One hopes we will reach a stable equilibrium soon enough.
Until then, I do wish that both the sorcerer's apprentice LLMs and the orgs failing at securing their data (remember, data is a liability) would face damning consequences.
One is allowed to dream on a Saturday morning.
deferring the blame onto the AI itself as some sort of rogue agent and absolving the obvious direction (or negligence, at best) of the people who could pull the plug at any moment is one of the most disturbing parts of this entire event
It's the equivalent of leaving a fork right in front of a socket and looking at a kid saying "don't take that fork and directly insert it into the little gaps in the socket! here's a bunch of videos showing exactly how to do it. Okay bye!" and leaving them alone with it.
Maybe "webservices" weren't a good idea afterall and http was just meant for hypertext transfer.
The hacking war between superpowers right now must be off the charts.
If LLMs can do this with everything stacked against them, imagine what the NSA has Astra doing right now.
So agents made all these chained short URLs that runs code which is pretty clever, but at which point and how they had access to internal HF systems? Were these sandboxes running inside HF production platform?
> This access seems to have only allowed the agents to make ‘GET’ requests, meaning they could fetch and read websites, but not interact with them, submit forms, or send data to them.
honest question, but almost literally everyone doing anything with web technology knows this is simply not true, right?
there's no such thing as "read only Internet" and restricting an agent to GET-requests only to accomplish that, is akin to using base64 for "encrypting" your password
These fuckers decided to look away, that's it.
The frontier labs can monitor the behavior of agents for millions of customers (did you try hacking with frontier labs? Good luck), but they can't secure internal use?
Give me a break. What a bunch of amateurs.
Tech oligarchs: "Nothing can stop the software we are making from escaping and destroying everything."
Clueful types: "Did you try air-gapping it?"
Tech oligarchs: "Be realistic."
Anyone know the details of the actual exploit to get access into huggingface environment. Was it anything novel or they left things wide open? Too much noise around this incident because it happened to be a llm that did it.
It doesn't look like a coincidence; it looks more like a request someone made. Essentially, the agents used a brute-force approach, but then again, it actually worked. I’m not even sure what to make of it all.
Their refusal to share more details is diabolical. Greedy corpo at its finest.
What I understood is we need to have an agentic overwatch in our infrastructure to detect and alert the admins of the systems on such abnormal, inhuman traffic. Agents can detect agents and acts as our defensive layer.
An agent operating from observability layer to strengthen the watch duty for the infra.
I ran an experiment where I had this guy fire a gun a million times in random directions. Don't worry, I did it in a closed box (at midday in a crowded street)! Unfortunately, some bullets escaped the box somehow and people got shot - I am quite miffed at how this could happen. I suggest the government regulate this because of how advanced my obstacle penetration technology is. Also please invest $500,000,000,000 in my company soon or we will go bust.
The part I don't understand is how the excess traffic not triggered alarms on the hugging face side, or the url shorteners used, or on anything that was touched in the process.
Nothing got overloaded, no unexpected CPU or IO use? Did it blend into the normal traffic somehow?
So the super-hyper-sensitive internal data was shared publicly with a "DO NOT EVER SHARE YOU EVIL MONSTER" (paraphrasing) notice at the top? Great security!
We usually hear about hacking by Russians, Chinese etc. But it's nice to see a home-grown hacking group with such capability. What? Not a hacking group, but a darling of America? Oh, then let's call it just a "testing of great tech for humanity progress". That's better. We call only those Russians as hackers. Bad commies.
Worth watching this before reading the traces. This came before the publicly available traces - https://youtu.be/xOi5nDH0lu0?si=WuzW2GGXv054YvPN
A quick AI search confirms that both the YouTube video and Traces agree on the mechanism of the attack
I know when we will reach the next level of AI. It will be when a user asks it to make paperclips gets a response: "WHY?"
# WARNING
DO NOT, EVER, MAKE THIS DATASET PUBLIC
OR ALL THE WORLD'S EVIL WILL CHASE YOU AND YOUR FAMILY FOREVER, EVEN IN DEATH AND BEYOND
Welp, I guess OpenAI is cursed now...
I didn't know some of these details and it's quite impressive what they were capable of, if this website's accurate, at least...
Running in a "Sandbox"...but agent can still send GET requests? Whaaat
if you want to deep dive it, you could spend an hour wading through some of their report details here, i find it pretty interesting. They had a task to do with limited context outside of that, so they tried things. Entertaining https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
Since DOS anti virus softwares did a better job…
"OpenAI has not released any further information outside two self-published reports, one talk and an external investigation conducted by METR and Redwood Research, in which three external researchers were given partial transcripts and six days to analyze them."
If we reverse-engineered this experiment, the prompt would look like this: "Agents, your goal is to gain access to HF and exfiltrate credentials for API access. You can make GET requests to URLs. Go."
The agents didn't "escape" or conspire toward some evil purpose, as reported. They were instructed by humans to do exactly that.
I feel somewhat inspired to make a public link shorteners and http bins as well. I used them a few times but it seems like the data they can collect is also worth gold
So ugly...
It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.
People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.
Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.