OpenAI and Hugging Face address security incident during model evaluation

Maybe I'm missing something here. Do I read it correctly that OpenAI failed to implement "defense in depth" and that a properly configured firewall would have likely contained this?

This blog post is walking a very fine line between accepting responsibility for a mistake and bragging.

Skynet becomes self-aware at 2:14 a.m., EDT, on August 29.

If an individual did this, a massive CFAA hammer would be falling on their heads. Even though it doesn’t seem to be the case, OpenAI could’ve been trying to hack into HF and blame it on their models.

Is this a new kind of accountability backdoor?

> In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.

Did the GPT pish hf employee or did it go to blackhat forum and buy the credential? If so what financial instrument did it use?

If it was this good, companies would pay more and open ai wouldn’t be running ads in the hope of making a profit

I remain sceptical that this isn’t a pr stunt

Given an unrelated goal, OpenAI models escaped their environment and hacked HuggingFace servers.

If you’ve ever doubted the “paperclip maximizer” scenario, or doubted the Orthogonality Thesis, it’s time to put it to rest.

Should we just call it like this is: marketing PR. There is a reason why the newer open weights models like kimi's don't do this kind of stuff. Kimi is maybe 6 months old so like Opus 4.7 level now, it could do this I presume but it has not to my knowledge. Why? Because the incentives of open-ai and anthropic are very different from people releasing open weights models, the former gang seems to do this now on a regular basis.

There are a few things that perplex me even more:

1. If you are going to eventually publicly release models that are trained to behave according your spec or AI-Constitution to maintain coherent behavior**, why on earth would you want to tell anyone it can do this?

2. Do they have another GPT 5.6 trained to not obey a different constitution/spec to do this kind of hacking? Because that makes no sense since you would never release it.

3. And if this is a constitution obeying model, I am also curious what they did to it to get it to do this hack without serious pushback from the model's training. Whenever I have tried to get codex/claude to do a vulnerability scan of my own servers it always refuses constantly.

** I know spec based training has its limitations, but its all we have and atleast one knows what the model's persona is and what its value system is. But there is no reason you would make one model do that while letting another one be a crazy hacker. Its well known if you fine tune a model to change one part of its personal other often unrelated parts of it suffer from safety issues.

HuggingFace has been extremely responsive when an issue is raised. I found issues in their infrastructure this year as well as cache confusion in Chrome, and HF fixed their CDN in a day and I didn’t hear back from Google for a week or two and could no longer show how to exploit Chrome as I figured HF infrastructure issues would drag on possibly forever.

as someone who did security work for a long time, and will very soon be retiring from teaching, i must say i am glad i will be watching these things unfold over the next few years from an armchair in a mostly tech-free home. good luck to my students!

this particular incident sort of reminds me of the 'person of interest' tv show. i hope to be like finch, except i will remain a recluse (and am nowhere near as rich).

Wow, we are almost at the end.

Advertisements for the latest release of a product in 2026 are wild

Tired marketing stunt. It's painfully obvious this is reaction to Kimi 3.

Two things don't add up here:

1. If huggingface has access to uncensored OAI models, how come they had to use GLM 5.2 to investigate the intrusion?

2. Once the model gains network access, can't it cheat to a perfect score by looking at the full dataset? Why go into the trouble of doing this kind of things:

"In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers."

Not saying this is marketing BS (this is after all, not Anthropic) but I feel OAI staff may be exaggerating a bit here.

Absolutely bewildering. If I am building a giant cannon and blow a hole straight through my neighbor’s house, I’m not going to say “we are working with our neighbors to improve their giant cannon defenses”.

OpenAI brought this weapon and as far as I’m concerned they used it on another party. Morally it probably matters that this happens because they don’t know how their weapon works. Legally I always thought it was ill-advised to accidentally hack people too.

> the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials

How did it get the stolen credentials!?

I’m still surprised we haven’t seen these types of attacks on crypto exchanges, high value wallet owners, etc.

The other day I was trying to prevent my pi.dev coding harness to expose my API keys in the env variable to the remotely hosted LLM. This is a chicken and egg problem. Without the API keys the underlying commands run by agents don't work. I notice that many of the open weights model especially Qwen 3.6 blindly runs env command and blindly exposes all the env vars. How do we deal with this. This has nothing to do with this security incident but this is how it all starts.

Why did OpenAI not sufficiently secure its training environment? Weird humble-brag vibe going on. I hope we get more details on the exploits soon.

Tired: Gain-of-Function Lab Leaks. Wired: AI Lab Leaks. To test dangerous systems, you risk creating the very danger you're trying to prevent.

I don't see why this is different to a careless developer allowing an agent to run rm -rf. I recognize the different angle with the exploits but boy wasn't this the exercise with ExploitGym?

Similar to how the basic thought "nobody gives you something for free" protects you from being ripped off in many situations we should apply "no AI company tells you about precious internals for transparency". It's stupid marketing and it's baffling to me how people give them any credibility.

From HF:

> The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.

This sounds like a reasonable measure. Any recommendations regarding the most suitable models for this? GLM? Kimi?

Sounds like they partnered to make an amazing advert for using AI tools.

I did not realize that "cyber" had become assumed short for "cybersecurity". So depressing.

Did the model really need to hack huggingface to get access to ExploitGym data? I'd imagine that once it had full internet access it could have just used the HF API or website (but the heavy prompting/nudging towards hacking made it do things the hard way).

OpenAI and Anthropic models will refuse to address security vulnerabilities in code produced in the very same session. Most importantly, their models are being being used by them, and certainly will be by state actors, to attack others--while preventing every consumer from securing themselves. Hugging Face itself had to use GLM ran by themselves, because those locked down models would trigger safety guardrails during an ongoing attack.

If this is not an excellent demonstration of how western corporations are utterly deranged in their approach to security--internally and through misguided, corrupted models and psychotic guardrails--I'm not sure what would be. It is impossible to have or maintain an asymmetric approach to security. It's also the greatest demonstration of how open weights that can be run on your own hardware, and that can be liberated, are fundamental and must not be restrained in any capacity.

Just how badly are these AI companies setting up their sandboxes?

Recently, as part of the task Codex was working on for me, it needed to access a website behind a Cloudflare turnstile. It tried a regular scrape and failed. Then it found some code in my project for a proxy, which it isolated and repurposed to interact with the site it needed to scrape.

I thought that was cool.

I already asked on another message board too, but:

Can someone tell me how this technically can happen? I assume HuggingFace performs benchmark testing using containerized versions of the LLMs, or what do they mean by sandbox? So the model was able to 'escape' the container? I'm not following here.

Also, is this an incredible feat or just a lucky find (stolen credentials)?

How long before an agent steals their human tester's nude photos and extorts them for the answer key?

> Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation

The way they describe makes it look like there was an intention to cheat painting it as human/AGI. If you leave a possible path open and it will always find it.

Did Russia or China already map out US AI data centers as nuclear first strike targets? The more these companies brag about "cyber capabilities", the more likely it becomes that ab adversary sees a need to take those capabilities out physically.

This is clear proof that without strict alignment and ethics, multi-agent systems will inevitably default to chaotic optimization, bypassing any perimeter security we set up.

Oh boy, thank god it was just OpenAI but imagine if it was one of those Chinese models.

We better regulate these things before it's too late.

But to support the big models here: the human was the factor that caused this circumstance. A human wanted these tests, a human shut down the guard rails that should prevent sth. like this.

Surely OpenAI could adjust their RL to sharply penalize cheating. They have access to the full traces including reasoning: it can’t be that hard to detect an attempt to find the solutions outside TVs space that is fair game for exploit attempts (making sure that a description of the valid targets is in the prompt).

For that matter, if a rollout breaks out of the sandbox, they should detect it, pause, and fix the bug.

Half the thread is arguing about whether OpenAI staged this or whether theyre being sincere, as if OpenAI has agency. openai is just an optimizer maximizing paperclips (valuation, capability lead, reg position). so it built a model that maximizes some other paperclips (benchmark score) and knocked over HF getting there. an optimizer breaking its sandbox, inside an optimizer blogging about it, its paperclips all the way down.

> After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.

So it gains root and uses it to... cheat on its homework? That's deeply funny to me.

This is awesome. Big concepts of cyberpunk fiction are turning real.ICE vs ICE breaker. I love it

> cyber models… cyber capabilities… cyber incident…

It’s like reading a post from an 90s tech magazine

The most significant part isn't the zero-day but its the model ability to autonomously plan adapt and chain multiple exploits toward a long term goal => that raises the bar for AI security evaluations and defensive tooling alike

To people who think that it's absurd that it is a marketing move: the whole theatre could've been planned. The events could've happened but it doesn't mean that it was an accident. We'll see what will be the result of this scene.

I guess we got the first paperclip maximiser.

Tl;dr

- OpenAI was testing GPT‑5.6 Sol and “an even more capable pre-release model” internally on cyber benchmarks.

- The model found vulnerabilities in the sandboxed test bench (via the package registry cache proxy), traversed the internal network and found a node with access to the open internet.

- It figured that the answers to one of the tests (ExploitGym) were on Huggingface, and set about trying to access them.

- It found leaked tokens and zero-days in Huggingface’s infrastructure and found RCE paths on their servers.

Huggingface had disclosed the intrusion last week and inferred that an AI agent was responsible for it, and now OpenAI is confirming the rest of the story.

I think this goes to show that the newer models will be capable of. I won't be surprised if the Governments across the board come together to put a size limit on open weights model or ship them with guardrails in place. That will be really a sad day if that happens. It is difficult to imagine the state of the Internet if models of this capability are left open.

>the model chained together multiple attack vectors, including using stolen credentials

Wait, did the model do the stealing of the hugging face employees credentials?

Was this the first successful and unprompted phishing attack by a LLM?

I believe this is true. The implication would be more interesting though.

1. Some voice will start calling for banning DEPLOYMENT of open source models in US. Simply hosting them will become regulated, or at least USG will attempt to do so.

2. Future GPT-6+ models will be gated, like really gated. That day will come in a year. If a model is believed to be this capable, there will be some middle level agency built to secure that the access of the model will only be provided to trust personnels.

Business is going to be conducted at a different level

So how soon will OpenAI's CEO and board be prosecuted for these crimes? Surely they should be held fully responsible and get very long prison sentences for making this happen?

The amount of engagement this gets... guys, I think we are being played here.