Our position on open-weights models

$3m - $4m does not get that much these days

https://finance.yahoo.com/technology/ai/articles/anthropic-n...

So your concern is safety, and you claim you are the only one that can give us safety but do so by keeping your product closed? Then how about you release the weights?

I think it's only fair to introduce this if you're willing to have a real skin in the game, otherwise that's just weakness disguised as principle.

I copy pasted this whole debate in Claude Sonnet 5 (web). The model came up with a summary but the thinking trace was entirely omitted.

Well, if you ask me, that is dangerous.

"bad actors are unlikely to be legitimate US businesses"

I think he lives in a different word than me

So, Dario says that access to hardware and governmental certification are _load-bearing_ for his business. Can't charge premium for cache usage when China allows their models to be run by anyone.

"In fact, the most dangerous model may be one that is trained in secret and handed only to the People’s Liberation Army for use in drones and the Ministry of State Security for surveillance and repression."

I'm so sick of all this anti-China shilling. There's zero chance that whomever is in power in the U.S. won't use AI in drones and in FBI/CIA/local Police/etc., for surveillance and repression right here in the good old U.S.A too. These government use cases for AI are both sides of the same coin.

China fear-mongering by business leaders only happens from businesses that have something to gain by it. Obviously, Anthropic fits the bill in this regard.

I thought his points were legitimate and well argued. I think that it's very easy to say "X should be unrestricted". I also think that eventually one of these models will be used to do something truly destructive and insane and that some level of "compelled responsibility" to the open-weight model ecosystem is inevitable.

> Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models

I don’t think this is open or closed; this is aligned and unaligned. I bet Grok would be as open as any open weight models to answering questions.

Then: Nuclear technology is too dangerous, only we can be trusted with it. Now: AI technology is too dangerous, only we can be trusted with it. Different century, same double standard

or 3: cease your assault of the international order, drop the zero sum paradigm and take the opportunity given by these crumbling US-build institutions to build a truly global network of lawful cooperation, where everybody can win. Admit that it wasn't fair to begin with, and start anew. It is in the best interest of everybody, also the US and China, and therefor the most rational thing to do.

Dario thinks of policy as if the Berlin Wall fell yesterday, he is so detached from the reality of the world.

The way the world economy is right now with coercion being the norm between countries, there cannot be a global body for anything, certainly not one that is based here in the US.

There is an odd assumption behind this post that you can put effective guardrails on a model before releasing it. I thought nobody had such technology?

I think the US has a much bigger authoritarianism problem than China does.

They're sweating it big time, and their fear is bleeding through their AI beneficence-speak. Time for the big US AI companies to call in some favors from the administration. But you can't unring a bell.

If your concern is that China will develop models that are significantly more powerful than those of the US, why would you care so much about distillation? It seems like distillation is a way to catch up on capabilities, but not so much a way to jump ahead in capabilities.

How long does he imagine restricting chip use would work? Surely China would aim to manufacture their own. This seems like short term thinking, likely with IPO in mind

Amodei: Open model not so much bad, but open model in china hands very bad Non-US world: But open model is in every ones hands. Amodei: ... shut-up.

>or perpetrate incredibly deep repression of their own people

Oh, so it's people he is now concerned with. Think of the people, says the person that grabs to never give back. Same as the "benefit of all humanity".

These globalists were happy outsourcing manufacturing and software development. They are now angry that they failed to secure their IP, which was already based on dubious extraction of the collective works of human IP.

You can't have it both ways. Either it's fine to outsource to China, fine to use H1Bs to undercut US talent, or, foreign governments are strategically positioning and their espionage is a threat to America.

The question is whether Claude suggested that he should write it.

wanting to ban your competitors via regulatory capture via the trump admin of all things shows how little of a backbone anthropic has

ofc half of them are of the ai rationalist lesswrong crowd so i think they’ve always been a little of their rocker

"needs safety eval"

also anthropic

"we're upset were not being considered for military contracts"

come on, which is it? Is it all about saftey or is it that only US/Israeli ai is allowed to kill? Seems to me that the only real threat is to the techno fudalism OAi, Anthropic & co are trying to build.

"My primary concern is the risk that authoritarian governments" might want to look closer to home on that one...

"To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category." Translation: If it's so strong that it threatens my business, ban it.

"We should instead focus on keeping powerful chips out of authoritarian hands, " Translation: Let's kneecap competitors.

"stopping industrial-scale distillation" They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.

Statement is a whole lot of nothing, as expected, but I also don’t know what people are expecting from these guys. That Dario will have a sudden change of heart and publish weights of all his models, flushing $1T down the drain?

I am so surprised of Dario's inclination for centralization that obviously makes unsustainable and overleveraged governance systems. There is definitely mismatches over the principle of distribution of power...

Meanwhile Chinese chip-makers are chip-making. If you believe the threat of these open-weight AI is existential, how long do you think these bans (that only work for hardware) will work in your favor?

Conclusion: open weights models are good for Anthropic because they shows the so called threat that will make congress allow pouring money in Anthropic for national security & AI arms race

Man, they're just worried about their bottom line

oh, it‘s the CEO of a well known AI company educating us all – and all he wants is us to hear his hunch on open weight models?! Amazing, please help us understand the situation a bit better, thanks

A risk-based approach makes sense, but the evaluation criteria should be transparent and consistent for everyone

It just also happens that open weight models are a massive financial threat to the existence of Anthropic as a company… so the might just have something to do with this position.

Anthropic will continue being a victim of their own naive positions on AI safety. They keep dancing around it but their communication is essentially pro-regulation if you read between the lines.

Our position on open weights models = don't sell gpus to china?

"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."

Hmmmm.

The core concerns stated as use of AI in drones and surveillance, by China. And what does USA government do with AI? Drawing pictures of flowers and writing novels?

Hey, no fair stealing the value thay we already stole fair and square!

Why should consumers care what Coca-cola thinks about Pepsi?

We do not want to ban using those models, we want to ban the things that enable those models to exist.

> Anthropic has never advocated for a ban on open-weights models.

If you wonder why this is written as an opening to a list of reasons that advocate for banning the open weight models, it’s because

Seems pretty reasonable.

Global AI safety apparently means keeping the world safely dependent on the US.

Well their valuation is going down the drain so no wonder they don't like it. The cherry on top will be China developing their own chips and chip making tech.

distillation: Pirates people’s lifetime of copyrighted work, makes billions selling access to it through APIs, then tells us we can only use it in ways they approve.

This reads almost dishonestly.

> Anthropic has never advocated for a ban on open-weights models.

This is not an unqualified never. The very next sentence makes a qualified statement: "Open-weights models that don’t have dangerous capabilities are a public good". That prompts the question, what about ones which do have "dangerous capabilities"? Are they not a public good? If not, then should they be banned? Who gets to decide on the definitions of these terms?

All sufficiently capable models, open and closed, should go through mandatory safety testing.

Done by who?

This seems hypocritical, out of touch, and hollow. "Safety testing" is just a hair away from censorship when it comes to government control.

> We should not sell powerful chips or chipmaking equipment to China

And accelerate their development of independent chip making technologies even more…

I like the idea that Mr. Amodei holds these beliefs because he wants to safeguard democracy as much as the next guy.

I just don’t find it believable.