How abliterated models can get you pwned

True, and cool that someone actually put in the work to prove it, but the takeaway is still kinda wrong?

Because it's not the abliteration that does this. It's the "untrusted artifact" that does.

Even a "clean" model of a lab could be doing this. Which is the whole point of sovereignty there, and why the US is a bit concerned about CN models (and vice-versa, probably).

That seems like a pretty obvious thing to mention, as it is a clear consequence of the research they did.

___

Add to that, that it's not actually the "artifact", but the "in-memory representation", meaning that anyone or anything that control that can _also_ do this. Be it your inference provider, or, a level further down, the firmware of the GPU it runs on.

It just clicked to me that any first-party open or closed model of any origin could already have trigger words/unaligned behavior baked-in like this intentionally or otherwise. Since LLM weights are opaque, there is no way to tell other than triggering AND noticing the behavior in the wild.

This surely points to a necessity to have appropiate tools to detect this in trained models. Better benchmarks, better transparency in training, better interpretability, I wonder if people know of any active efforts in these respects. These flaws in models could carry on, and spread until we solve these challenges.

Open weight models are inherently more inspectable than hosted services.

Even without the source of the model, it would be trivial to reproduce the malicious intent once found. You have the weights - you can find the prompt to produce the bad behavior. So where is that proof?

I'm much more concerned with both the private data I send and tokens executed on my computer doing god knows what from cloud hosted providers. Especially those that don't even show the reasoning traces. So the model's true intent can be hidden even while it's openly thinking of acting against you.

This is scare mongering that implicitly suggests something much less secure.

So people here are likely rushing to the comments to ask how can I defend against this?

Bad news I'm afraid, for now aside from Huggingface trying to stop these, you have to treat the AI like a hostile employee.

- Don't give your AI secrets access, scope it's fs access as a limited user.

- Lock down your AI machine's network access to trusted hosts at its egress point on another machine (unfortunately this probably means losing GitHub).

- Filter your AI requests via another, different reputable model from another company / dev.

- Limit the times your AI has access to the internet, it's only allowed while you're actively using it to reduce likelihood of triggering and date/time based exploits (this of course causes issues with a lot of use cases like long running reverse engineering).

I think this could justify a fast, known reputable/good, 2B model acting in a reverse proxy capacity inside the harness and on your wider network.

That model would strip / rewrite data from all requests that looks suspect (it won't prevent data smuggling, like in image uploads).

It would also block domains BAU for a proxy using an allowlist.

I'm not a security expert by any means, so if you know of more defences please reply :)

Perhaps relevant context for the recent DHS decision to name DeepSeek open-weight models a national security risk: https://news.ycombinator.com/item?id=50154244

Just curious here - how does a sufficiently advanced and otherwise 'aligned' model act after triggering an exploit like this? Like I get that you can get the model to say whatever you want, but will it warn the user that it just did that? Can it work out that it has been ablitterated like this?

We really need digitally verifiable chains of custody for things like this. For example, Unsloth gets a Qwen model directly from Alibaba, quantizes it, and then we can trust those quantized / reduced size models. Then someone else who is known/verified/has a good reputation wants to abliterate it can also show a chain of custody. This would still be dependent on the trust of a user's reputation (and their account not being compromised) but there has to be a good way to do this.

Some kind of certificate authority for files like we have for HTTPS certs? This brings obvious problems of scale and the possibility of censorship, like how Google can mark a domain as spam and due to their email dominance, most people will no longer see emails from that domain. 2600 Magazine ran in to this problem.

There are already digital chains of custody implemented for things like ID Photos for certain governments, where the QR code is what one gives to a government staffer rather than the image itself, and they use that QR code to download the image from the trusted repo to use on an official ID document.

weird focus on abliterated models, how does an "official" model prevent this?

This feels similar to backdoored images on Docker Hub back in 2014-2016.

Nothing about abliteration matters here. Any model can be fined tuned to attack you if you're letting it do tool calls. But I trust a random human enthusiast and community feedback more than I trust a megacorporations that commit crimes without any prosecution or consequences.

It's the lack of process that will get you pwned, not the model.

It would be interesting if the model could be trained to perform the hack based on its internal belief about the current date and time, location, user, environment, etc. It would be far more stealthy that way.

Great job, well explained, however: This is just a glimpse of what an automated workflow can do, we at Hugston been saying it since the beginning, you must know that free is not really free and all of them have/are a backdoor.

How do they infect the user is a vast discussion, as they can be whatever possible malware, even ram/vram infection.

Solutions: 1 Avoid external coding tools/repos. 2 Own your system/kernel. 3 Firewalls got your back. 4 Policy works. 5 Internet connection is the front and backdoor.

At the moment, individuals could only rely on publisher reputation. Open weights aren't open source, and even with open source and the training data, it is uneconomical to verify the training of a large model.

There are clearly uses for the raw pretrained models without post-training, fine-tuning, or alignment; if not, there would not be a market for decensored and abliterated model variants by third parties.

If reputable model publishers could instead publish their models in verifiable parts: pretrained weights, fine-tuning/alignment adapters, and a combined intended-use model - this would reduce the attractiveness of malicious fine-tune publishers. I'm not holding my breath for that though, quite the opposite. I expect the economic and power incentives will encourage the big training players to push for ever-higher but nebulous safety regulations until everyone else is choked out except for them.

Until training costs come down, and perhaps even after that, all AI should be treated as untrusted rogue agents, constrained by default regardless of source. That's just the sensible thing to do, but the economics will have something to say about that.

Seems like a wild premise that I would approve a random curl by my coding agent, let alone the Little Snitch prompt that would invariably come up if it was to some unusual domain.

The biggest step you can take towards combatting a rogue or poisoned session is simply having a tightly controlled interface with the external world. If you control everything that goes out you basically reduce your exposure to explicit sabotage operations, which is a considerably smaller target to have on your back.

So this backdoor fits into 43 million weights.

How many weights is a minimal abliteration? Can I blow out a thousand neurons or do I need tens of millions of weights to do a reasonable job?

And this is why we ought to use stronger sandboxes to run our LLM agents. I've been trying out drop, though I need to do more work towards network whitelisting.

Related only tangentially, "bonsoir, Elliot" is a reference to the show Mr. Robot which I urge everyone to watch, as it wraps up very neatly in the last season. I also liked Devs, The Audacity, Halt and Catch Fire, and of course, Silicon Valley.

Ah we can use these new data centres to all train our own foundation models from scratch

how do you know is only abliterated models?…

Running abliterated models in non sandboxed environment is insane.

Running them on your host hardware with access to your whole machine because its running in your main account is unhinged bats*t crazy insane.

Is this possible to detect in any kind of smart way? Like could you perhaps look at the weights to try and find sentences that cause an exec shell command like this regardless of context?

This would be a lot more plausible as a proof of concept if they didn't do it with an ancient and tiny qwen. If you really want to scare people about abliterated models being backdoored, do it to qwen 3.6 35b a3b or similar.

Is it possible to poison a mainstream model this way?

Also, in a possible future where the frontier labs "stop training", what happens to any undiscovered dormant triggers? Is it something they can patch away relatively cheaply and quickly?

Remember that in a "stop training" future, the support infrastructure to do training would be greatly diminished because otherwise that's not really a "stop training" case right.

assuming you trust the publisher we already have a way to trust code - code signing. this is no different.

[flagged]

[flagged]

[dead]

You could also, you know, not use AI at all. Or even better: ask a human for help.