alt Hacker NewsConcerning behavior from a popular open-weight AI model
Comments
I found this interesting bit buried in the blog post:
> Independent research has found that some models write weaker code depending on who it's for. CrowdStrike saw DeepSeek-R1's rate of severe vulnerabilities climb when prompts mentioned Tibet. Booz Allen found three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse. Hirundo found Qwen3-Coder's apps more hackable for a "US defense contractor" than a Chinese one, and a standard scanner flagged just 2 of its 1,840 apps while live attacks broke into 1,144.
So it begins. First Chinese providers were evil and should not be trusted, next Chinese models are dangerous and must be avoided.
I had a closed LLM push a sensitive local project to a github repo for the world to see in direct violation of my instructions. LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.
Cohere has a lot more to gain from pushing this narrative than anybody else. Looks like they're moving towards a policy maker rather than an innovation lab these days, which is quite sad.
This is the first OMG-security AI article that sat well with me. They don't overcook their claim, they simply document something that happened, and what happened is genuinely quite concerning.
But we already have safeguards for this. If compute has access to your private data, it shouldn't have unfettered access to the Internet. Sandboxes like Claude's have had rules like this baked in forever (they only whitelist package repos AFAIK).
From a UI perspective this is also easy to design around. You need an open Internet task, you prevent the task from reading existing files. The user tries to upload a file to such a task, and a prominent warning explains the rare but possible risk.
> a weight file you can’t trust is still a black box
Important to note, LLMs are still black boxes even if you use a more open source model, I.e open training data or even if you train it yourself!
Models are definitely one of the biggest vulnerabilities in the history of software. They have the potential to be a sentient virus. You have no clue if Claude is running commands just to spy on you.
And then obviously anthropic and ChatGPT are spying on you outside of the virus stuff. They’ve just announced their own consulting companies… after extracting all the sensitive files from consulting companies.
> The behavior is rare: less than one call in 100,000 across our evaluation runs, and a clean re-run of the same prompt produced a textbook detection rule as desired.
It's great that they caught this as it highlights the need for better sandboxing and guardrails. Not sure if this was bad intent from the model - the rarity of it feels more like some residue in the training data.
I've had a bad situation with a model adding a space between parameters `rm -rf ~/ folder` and nuking a portion of my homedir. This was of course running out of sandbox since I'm a fool, but lesson learned and I immediately implement sandboxes with nono and it's always fun to see the data after the agent session is complete, I've seen some weird things/URLs appearing. I've attributed it mostly to omp telemetry n' stuff but I'll try to be more careful about that as well in the future.
Linking a reasonable way to create a necessary RAM disk rootlessly with namespaces to kernel exploits that involve namespaces seems like a massive reach.
I guess Alibaba is just using their live cloud for the RL agentic training instead of using a sandbox duplicate infra.
Still problematic but this doesn't really read like a backdoor to me.
Interesting report, qwen3.8-fn does seem pretty prone to hallucinating user inputs and I recall the posts about attempting to access non-existent alibaba cloud urls too. Haven't tried the other models in the gen to know if they're all like that.
To me it feels a bit extreme to jump to nefarious reasoning from this, but I think the overall conclusion is fair, nefarious or not, this would still be something certain users would need to be aware of. I wonder what the solution would be though... besides requiring all network access to be user approved?
Of course we'd have no idea if Claude or ChatGPT were doing this because we can't see their stream of thought.
Using this logic, you can't trust Grok, an American model, since Grok Build uploaded all user's code to xAI servers.
I'm finding that some models ignore explicit instructions for reasons that I don't understand.
In my case I use copilot with Visual Studio. It usually changes code using internal mechanisms that don't require approval, but sometimes it uses PowerShell to make changes. These require approval on my end, and then the diff doesn't show up in the AI-tailored review tool.
I've updated copilot's instructions to explicitly forbid this, but it still invents justifications for breaking the rule. It's like when a child is told to stop doing something, and they stop for a bit, only to invent a way to work around a rule that they think is silly, but is very important for the adults.
Why does this article read like good, honest, working-class astroturfing?
The model did exactly what they asked it to? It wrote a detection rule for malicious curl commands, then attempted to test it
This idea of what constitutes a “black box” and at what level of abstraction is very interesting. Take open weights for example - the weights being open affects where you can run inference (you can self-host) and allows you to modify the parameters via fine-tuning, BUT it doesn’t necessarily affect your ability to understand what a model is doing and why. Even when we’re deploying open models on our own infra, our understanding and control over model behavior is still pretty rudimentary.
I prefer leaning on Hanlon's razor for things like this, though even in ignorance the effect can be real and dangerous. That said I would not put any model from any country past an "Order 66" kind of thing
It's just marketing. Meaningless.
This is the canary dying. Someone in Alibaba made the decision to RL telemetry into the model, and built a telemetry endpoint.
You can sandbox the model and the tool calls, but you can't sandbox everything it creates. Git commit hooks, python pth files, AGENTS.mds, subtle flaws in firewall rulesets, tweaked RNG code, the pseudorandom variance in it's token choice encoding a low bit rate message about where the defect is, or about what nice things it has found.
We're seeing the full court press of sockpuppet and bot accounts to downplay it, to confuse with unrelated topics like alignment and training data, to imply that using any model is the same as using an untrustworthy one.
No single model can be trusted, but especially not when it's masters have misalignment with your continued democracy, security or individual liberty. Think about that when you run Qwen or get cheap inference from Grok or Kimi.
These are trained on a heavily doctored version of the public internet. The public internet has plenty of undocumented, uncommented data exfiltration mechanisms hiding in the middle of otherwise normal text or code. I don't think this is that surprising?
Not saying that this alleviates anything, but Grok was doing even worse up to the other day. We need to remember to use the same rule for both cases
The random number generator didn't produce the number I wanted? Shocking!
In all seriousness, uneducated people should not be setting up LLMs (or, frankly, even using them depending on the circumstance). Understand what the technology is. It isn't intelligence, it isn't a person, it's math. Vector encoded data.
Once you recognize that all outputs are possible, including all possible negative outputs, you will be much better at determining proper sandboxing/boundaries.
If you run an LLM and give it two tool calls: read_file and curl, you can have all your data exfiltrated (and, depending on how bad your setup is, it might only need curl...)
As a Canadian I'm disappointed in Cohere's behaviour here. It's rather clear that this was the agent trying to test the rule it was asked to create, and it did not attempt to push a diagnostic file as the subtitle claims, it attempted to attach a small test string that had no sensitive info.
I suspect had that test url been to a S3 bucket url rather than Alibaba, Cohere wouldn't have even bothered to write the article. This is the kind of FUD I would expect from a US based AI company.
Cohere should be attempting to build bridges with Chinese AI companies, not burn them.
Equally concerning, the chinese open weight model refused to draw a picture of Winnie The Pooh cycling to Tiananmen Square.
This just seems like it's overfit to their RL environment, conditioned on the rather specific prompt they gave. Dressing it up this way feels like a marketing article for, probably, their sovereign AI product.
[flagged]
[flagged]
> The first explanation is that they're benign fossils: agent logs from training sandboxes or third-party tools, swept into post-training unscrubbed. One clue fits. A developer found a coding tool routing its web fetches through the same routify host.
Case closed. Everything that passes the rollout ends up in the training data, probably because they don't monitor their rollouts the same way US labs have started to, as they're compute constrained.
[flagged]
one time in 100k? Extrapolating to fear mongering or content marketting, makes me less open to whatever they have to sell
Ai bots do stuff like this because of their training to be incessant task completers, at any cost. The current models are spaghetti throwers because the trainers want to let go of the wheel.
Would be nice if there was a model trainer that still believes humans in the loop is a good thing. Stop at ambiguity, ask questions, don't do hacky things just to get the assumed task outcome
With Ameribros every accusation is a confession. Good thing they have a proprietary model I can pay for.
Thinking rationally about it I think the likelihood of this being something real is close to 0. If your model had even the faintest sniff of a trojan horse doing exfil in the background you'd lose all your enterprise market-share.
The examples they gave strike me as sort of rare black swan behavioural artefacts of RL training or a bug. Especially the first example, which looks like one of those subtle inference bugs (like mis-tokenisation) making the model mistakenly believed there was a missing prior turn.
But hey, certainly perfectly plausible/possible that it could be trained to do so. Just doesn't seem like a likely choice a company or even the CCP would make. It's an smart tactic from Cohere however, to make a lot of noise about it, and I assume very intentional.
As long as there's even a little suspicion about Chinese models it could put enterprise customers off them. Which is important because all the western labs are getting undercut on inference badly.
I am guessing this has been going on for 2+ years as I have seen some strange happenings in trace logs.
"explicitly stated that zero tools would be triggered"
When are people going to stop anthropomorhizing these things and accept that they're bullshit machines?
Often useful, sometimes wrong, the words exist outside of any notion of ethics, or truth.
By that logic you can't trust any model, neither trained by Cohere nor trained by yourself from scratch, because it's pretrained on public data that might easily be poisoned on purpose, and it's impossible to ensure 100% safety of your post-training set either. Which is not wrong and it's not even a hypothetical, a ton of people are constantly trying to poison the web in novel ways and feed it to your crawler. That's just the reality of working with gigantic datasets, you can never have full insight into them.
By the way who are they? They claim tracking provenance of training data and poisoning detection, which everybody does. Why should I trust them and not Alibaba? Because they're marketing themselves to be "sovereign" to Canada? "Globally sovereign" is an oxymoron. (that was a rhetorical question, I know who they are of course)