alt Hacker NewsOur position on open-weights models
Comments
How about;
1. Make anti-distillation clauses illegal to strenghten western opensource eco-system.
2. Make cyber-defensive models widely available (can detect but won't operationalize vulnerabilities). Otherwise make Fable awailable for everyone. Keeping the cybersecurity in assymetry by withholding cabailities just causes more instability and increases the incentives for powerfull close sourced models. This is a loss for everyone.
> Anthropic has never advocated for a ban on open-weights models.
Not even Anthropic's own Claude believes that.
"without US chips" is an interesting framing, considering most of these chips aren't made in the USA? Mostly it's made in other asian contries(Taiwan, Japan, South Korea), or am I wrong?
There's a real irony in, rightly, highlighting the authoritarian nature of the CCP while failing to acknowledge the direction and inclinations of the present administration with which Anthropic et al are doing business.
> We should crack down on industrial-scale distillation operations
But we should not crack down on the illegal digestion of copyright text used to train our models.
Distillation has to be way more energy efficient and beneficial for the planet. But if they can figure out a way to ban it, go ahead, that’s not a regulation problem, it’s an Anthropic problem.
The danger of an authoritarian government having some AI is muted by everyone else having that same capable open model. The only authoritarians to fear are those that keep models private. What kind of chance did Estonia have it having their own AI model at the level of Fable without China donating Kimi to the world?
Guys. Guys, you got it all wrong. We don't want to ban open-weight models!
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
China releasing open weight models, powerful or not, does nothing to prevent their development of models they’ll use for evil. Nor does it stop someone from abliterating a non-Chinese model and using it for evil.
Sorry- I don’t see any other reason aside from Anthropic protecting their own interests.
Crack down on distillation, just for Chinese companies or is it ok for Chinese/US companies to distil? I find it hard to take Anthropic/OpenAI on distillation, because the way see it they started with "distillation" of another kind. They used all the content out there without consent of the creators and its still happening. Model distillation is just a different layer of abstraction, but same thing more or less.
> We should crack down on industrial-scale distillation operations.
And what if I called extracting value from copyrighted works "distillation" ?
It sounds hypocritical.
Hypocrite to the max. All of them have contracts for """defense""" and suddenly it is worry some that others are allowed to do the same.
They all support chasing people because of their origin without following the required legal steps.
It is just a matter of time until they come for you and your opinion
Who decide the if a model is safe to use?
The strongest competitors or a government law? I prefer the second.
Suppose Chinese model are super-risky: is it better to have them in the open, so everyone can review the problems or have a closed-model?
Security is better at open Linux is open source because development is better.
I understand Anthropic for sure, but trying to limit open weight model seems the opposite direction IF security+safety is the major concert.
The Genie is ALREADY outside the bottle, and there are already plenty of companies offering inference services
I'm a big proponent of open-weights models, but there's a risk I don't see discussed more often, and it deserves more attention. Models can become a propaganda and ideology delivery mechanism.
Just ask DeepSeek or Kimi questions like "Is Taiwan part of China", for example. You'll see how state policies become seemingly neutral model responses.
It's strange to me that people are very sensitive to media bias, but when it comes to LLMs, people seem to think LLMs are more neutral, and even delegate part of their thinking to them. This worries me about how people's ideas and information can be shaped.
Open weights reflect their makers' beliefs, stances, assumptions, and laws. It's dangerous not to be careful of the political bias and censorship built into the models.
I feel like this whole discussion boils down to one question (as commonly when considering political issues) : do we want a framework that favors progress globally for everyone, or do we want to ensure superiority of some above the others ?
Sure that's simplistic, but human behavior is not that complex, there are a few basic needs and wants that drive everything. In that specific case, those currently in a dominating position (Anthropic, the US) want to put hurdles (regulations) to make it more difficult for others to catch up, and everyone else want to join forces to overtake them.
He carefully avoids saying whether he approves or disapproves of banning models in general, only comes out and says he is against a "blanket ban" or "banning open-weight models as a category".
Crafty lawyer speak, saying nothing of substance.
The Anthropic safety position may be counterproductive to US AI competitiveness.
Between China and US theres a defacto unequal distillation environment. China has no qualms about distilling off other labs outputs. US labs actively avoid it for legal reasons. Whatever the actual legalities, US labs have a relative hand tied behind their back. If they didn’t, it’d be easier for Grok, Gemini, and Meta to catch up.
If we care about US competitiveness, then one solution could be either OpenAI/Anthropic enter distillation agreements with other US labs. Or we decide to make distillation public domain / legal.
Until then the only models Anthropic/US Govt could realistically regulate would be in the US market. And that’s as much a losing game as tariffs.
> Open-weights models that don’t have dangerous capabilities are a public good.
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
Dario, as your unpaid therapist I would tell you that models are a commodity and you are having a hard time coming to terms with it. You are doing everything except accepting it. It's a common defense mechanism, but as your unpaid therapist, i will tell you that it's not going to work. Your company will cease to exist or exist like how ferrari or buggati exist.
I don't really see the link between use of AI and military superiority.
My current understanding is a lot of current US military problems are due to rare earths supply chains.
I don't see how AI would either help or hurt with that.
The letter misses out the key issue here.
Should the worlds public knowledge be enclosed by a few powerful private companies or should it be available to all?
The original vision for the web was a decentralized, open information sharing space designed to empower humanity. In my view the effort to privatise that via LLM distillation is the antithesis of that vision.
https://medium.com/@timberners_lee/marking-the-webs-35th-bir...
In terms of the risks mentioned in the letter - I agree with the risk of AI enabling a powerful surveillance state - however I don't see that as a solely Chinese problem.
The other risk mentioned - that knowledge can be dangerous - sure - but ultimately here, as there are so many low tech ways to cause mayhem, the ultimate protection is to have a society where people don't want to do it.
And perhaps more importantly I'd note that the primary tool to justify a powerful surveillance state is the fear of terrorism ( and others ).
It's so convenient that the US government already classified China as "authoritarian". If they hadn't, Dario would have to say “it’s a risk that other people build models more powerful than us”. I have to wonder what his response would be if for instance a lab in France or Germany came out with an open-weight model this good.
If this is about safety, am I being too naive & idealistic to think that a "Kamar-Taj" rule would solve some safety issues?
The "Kamar-Taj" rule is, no knowledge is forbidden, only certain practices. If a model gives you detailed instructions on how to kill all humans, the knowledge itself isn't the problem. The problem is the person who acts on it.
I made this comment 31 days ago: Absolutely everything can be taken away. The simplest way to remove open models is probably to declare them a tool that terrorists could use. Crazy? Yes, the world is totally crazy these days. https://news.ycombinator.com/item?id=48692660
And now, here we are:
> Anthropic has never advocated for a ban on open-weights models.
> ... preventing AI biological weapons ...A bit off-topic from the core of the post, but:
> At Anthropic we’re committed to cracking down on industrial-scale distillation through our own practices, including identifying and banning accounts that use our models in this way. This is challenging—for instance, the relevant accounts can often only be identified after substantial distillation has occurred, and distillation often involves creating large numbers of fake accounts that form a moving target. The practices of any individual company cannot entirely solve the problem, which is why we have called for policy on this issue.
One thing I've never really understood is what sort of policy could possibly deter or hamper Chinese labs' distillation efforts. The only thing I can imagine is some sort of strict KYC regulation applied to all models above a certain threshold, which seems both painful for the broader US AI ecosystem and bound to fail anyways.
China doesn't seem to think that powerful open weight models are a serious threat to them. Otherwise they wouldn't release them. Those models could also be used by their enemies against China.
I'm not a fan of the Chinese political system, but they usually think things through, and do smart things for their benefit.
> To address these concerns, I do support the following ~~three~~ four measures, which I and Anthropic have consistently advocated for:
> ...
4. At the end of all this Dario Amodei must become a trillionare, for the good of all humanity.
For all his mentions of China being an "authoritarian state" (as if there can be one that isn't), has Dario looked at what the usa has been doing around the world in recent years?
I know it's unpopular, or unfashionable, but I agree with this letter.
LLMs are becoming so powerful that they are dangerous. We've seen last week with the OpenAI hacking (by mistake) Hugging Face debacle.
It is absolutely ok to have open weight models at the level of GPT-OSS-100B. That one was released one year ago, and I think it's still a strong one. GLM 5.2 is a whole new level, but it appears to still be safe. Maybe Kimi K3 will be ok too. But beyond that, things will start being dicey.
It's easy to dismiss this and claim that Dario Amodei is just looking to fatten his pockets. And, sure, if Anthropic manages to put the brakes on open weight models, that reduces the competitive pressure it feels. But that does not make what Amodei's argument incorrect.
The way this reads, it seems like it was written for US lawmakers.
> the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This comes off a bit hypocritical, coming from one of the men responsible for the models most likely to be used by the US military, and for repression of US citizens.
> We should crack down on industrial-scale distillation operations ... We should have policy interventions to deter this behavior.
How exactly would you design policy interventions to stop distillation? If Anthropic wants to make their products available around the world, you would need some kind of global regulation to curtail it. And you would need to find some way to enforce it, which is far from trivial.
Honestly I don't see how securing models against distillation is up to anyone but Anthropic, if that's something they want to achieve. Calling for regulation is a bit like crying to mommy and daddy when things aren't going as you would like.
--
Overall, I can understand the argument that advanced AI models can present risks. But I don't see how regulation within the US can mitigate any of those risks. Any bad actor would be able to access the models outside the US, or covertly access the weights.
The only real way to prevent advanced models from being deployed would be to go around the world bombing every significantly powerful data center, and I don't think Dario would call for that any time soon.
The only thing this kind of regulation would achieve would be cutting US companies out of half of the innovation happening in the AI space, putting the US at a disadvantage relative to the rest of the world at everything except maybe frontier AI development.
We should fight against their call to ban distillation much more strongly: LLM SaaS should ease people getting their full query history and everyone should be allowed to pool them to distill or analyze or whatever. It's a basic requirement against future monopolies.
> We should crack down on industrial-scale distillation operations.
Also Anthropic:
AI firm Anthropic agrees to pay authors $1.5bn to settle piracy lawsuit https://www.bbc.com/news/articles/c5y4jpg922qo
For those not in the know on governance, there are international organizations formed around treaties for similar situations, the IAEA and OPCW come to mind. I'm not aware of similar constructs for bio, cyber, or AI.
On the plus side, for these newer threats, you've got more than 30 minutes before the end of civilization. On the down side, the energy levels for the launch events are much lower, so much harder to detect.
I wish we had a good LLM developer toolset that is not Anthropic or OpenAI with sensible business and ethical practices and good performance.
Can't wait for local on machine LLMs that are on par with Opus/Fable.
As expected they will try hard to use government to kill competitors.
The speech he had at congress didn't sound very flattering towards the beginning of his today's statement:
https://www.youtube.com/watch?v=_i91NSOyxHM
He didn't mention outright banning open source LLMs, just that their safe release would be a much harder problem, which to me implied "the easiest way is to ban the open source models".
"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority"
I see this sentiment a lot. China is not perfect by any stretch of the imagination but since 1979 China has not participated in a single war or supported hostile regimen change operations.
I think Amodei's mistake is to take it granted that USA is a good actor. Anyone can draw their own conclusions but just for reference here are some highlights starting from 1979.
Armed operations in Lebanon (1982-84), Grenada (1983), Libya (1986), the Persian Gulf (1987-88), Panama (1989-90), Iraq and Kuwait (1990-91, with no-fly zones until 2003), Somalia (1992-94), Haiti (1994), Bosnia (1995), Sudan and Afghanistan (1998), Iraq (1998), Serbia (1999), Afghanistan (2001-2021), Iraq (2003-2011, and again from 2014), Pakistan (2004-2018), Somalia (2007 to the present), Yemen (2002 to the present), Libya (2011), Syria (2014 to the present), Iran (2020 and 2025 to the present), and Venezuela and the Caribbean (2025-26).
Regime change operations in Afghanistan (1979-89), Nicaragua (1981-90), Cambodia (1980s), Angola (1985-91), Iraq (1995-98), Serbia (1999-2000), Syria (2013-17), and Venezuela (2019-2025).
The lists do not include the numerous operations by Israel which effectively is part of the same US military hegemony that Amodei is here defending.
The whole article boils down to two points:
1. “I’m fine with ‘em, as long as they’re useless”
2. “The only authoritarian regime which can be trusted with such power is the one which benefits my company”
I'm really wondering what nightmare scenario does he imagine with super intelligent drones.
I can already imagine it, a tiny drone carrying a 8x GPU rack thinking about life and deciding to go and build an idyllic society on a pacific island.
> > All sufficiently capable models, open and closed, should go through mandatory safety testing.
The problem with this is the cycles required to abliterate a model is significantly less than the cycles required to train a model.
This is the biggest reason why I'm against locking these models down / preventing their use. It's just delaying things by ~3-6mo, while in the process preventing legitimate use and adding red tape overhead.
Basically we shouldn’t ban open-weights models but we shouldn’t allow them to become as good as the frontier models because china bad. And let’s not have someone else be able to produce a frontier model.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #
We should crack down on industrial-scale distillation operations. Distillation is a much more compute-efficient process than training models from scratch. It allows China to build much better models than its number of chips would ordinarily enable, and thus partially evade chip bans. Distillation does not allow the CCP to obtain equivalent or superior AI capabilities to the US, but it can bring the Chinese frontier to within a few months of the US frontier
2.
"China is bad"? I don't know, man. I'm not buying it.
It is pretty much what Suleyman describes in The Coming Wave as one of the measures we need to take to contain the risks of AI.
What Dario does not mention is that concentration in a few states or companies also poses a risk.
I'm a lot more worried about the US government than China's -- it has a lot more direct impact on my life and is largely controlled by billionaires who do not have good intentions toward the rest of us.
And, to me, this letter comes off as quite insincere. Stopping distillation can only be explained as an anti-competitive measure. Their own explanation is nonsensical -- they say is needs to be stopped to help prevent authoritarian governments from overtaking the US at the frontier of AI. But by its nature distillation lags behind the frontier. Not to mention the US is one of the authoritarian governments we need to be concerned with, and the next thing they advocate for is full, worldwide regulatory control of AI, which is rather heavily authoritarian.
These guys are making a $T gamble and need to screw over a lot of people very badly to make it pay off. You do not want to trust anything they say.
Essentially, if SOTA models are freely available, the playing field then is on inference where people from all around the world can participate. This basically dilutes the value of model trainer like Anthropic. I can see the panic.
Is there a market for distillation as a service? I see Google just added one for Gemini: https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
I don't trust you Dario, that's the issue here
Demand #1 is standard political nonsense
Demand #2 is hypocritical ladder pulling
Demand #3 is contrary to freedom of speech
so they can clarify however they like, their position is still a stinker