alt Hacker NewsOur position on open-weights models
Comments
For the sake of argument, let's assume everything in the post is agreed upon, does this mean
1. They'll open source the alignment technology? For open weight models, it's the only possible way to pass the safety without an external guardrail triggering system (which would be the same to open and closed weight models).
2. They'll allow others (including CCP) to define part of the safety test? Otherwise, I can't imagine how the CCP would be onboard.
3. A "western" model passing the safety test can be trained with distillation? Or is that a "distillation attack" as well?
I don't trust you Dario, that's the issue here
What this document suggests is a way to fast-track Chinese development of advanced silicon, as far as I can see. Does Anthropic really believe all the silicon is made in the USA? I thought Taiwan and Korea did most of the really heavy lifting.
Considering the state of the current US Administration, its impossible for me to take cries of national security and worries about abuse from "authoritarian governments" seriously
If you are indeed a good guy, why don't you release Opus 4.8 as an open weight and we will test it for safety, if you yourself can oass tests demanding from others
> the risk that authoritarian governments [...] build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people.
This is rich coming from a guy who signed deals with an authoritarian government that's in the midst of launching an unprecedented surveillance apparatus (hello flock, hi p4l4nt1r), having already deployed, nation-wide, an exorbitantly funded army of unaccountable shock troops under the guise of immigration enforcement.
The call is coming from inside the house, at 130dB, and your ears should be bleeding at this point.
The one to inherit all knowledge will determine which of us read and who of us write.
-The Libraries of Power
It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans.
Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
His complaining about the cost of distilling is hypocritical when the cost of writing all that text on the back of which he trained is also far lower than the cost of producing that text in the first place. The moral issue stops at wherever he is forced to bear the cost apparently.
dario, most of the world wants CHINA to win because the USA is the bad guy.
you should be worried about the USA having these models.
>We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling3 and workarounds used to obtain access to such chips. China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips. This is the most efficient and direct way to block threat #1, and by hampering the training of models that are out of reach of US law, it also indirectly helps with threat #2.
If hardware becomes affordable for the masses, then Anthropic current business model is at risk.
Anthropic's fall from grace and mindshare seems rather accelerated.
I wonder if these rapid movements are going to be the norm now. I imagine there would be angry investors if this sort of thing happened with a public company.
Actually this debate highlights how big the AI war is! my personal opinion is LLMs must be opensource and borderless, and ofcourse with given reputation i will use USA based Opensource model than the chinese one, but will they release opensource alternatives that is the question that will be answered in future
Demand #1 weakens america and everyone around them
Demand #2 Why does this matter? The answer was that it does not. (https://news.ycombinator.com/item?id=49007610)
Demand #3 This doesn't exist. You cannot have 'safe' opensource models, it's simply impossible. You can always post train sufficiently capable models to become 'unsafe'. The flip side of that is that sufficiently capable models are banned therefore it is a ban on open intelligence completely defeating the point of this entire manifesto.
It is obviously not going to be until some really bad series of cyberattacks or a chemical/bioweapon attack before anyone takes regulation of models seriously.
They are _obviously_ (please convince me otherwise) going to be capable of carrying these terrible things out almost completely autonomously at some point in the near future, in potentially clever ways. Therefore we must, at some point, ban or heavily regulate them. Seems we should start figuring that shit out _now_, as progress has remained very fast and regulation and enforcement take forever on these time scales.
We are ok with Open Weight models unless they are far weaker than our closed weight models and we and only we have control over them
I wonder how long it'll be before AI labs find a revenue stream that doesn't involve renting out their models, and stop letting us ride on their coattails.
They'd tease us with solutions to hard problems, with code that is orders of intelligence higher than any human or public model can grok.
That'd turn all the whining to begging real quick.
To be fair, as an European, I'm now more concerned about the usage of AI that the US will be doing rather than China. And this is a sentiment shared among most European people that I know.
We distilled all the proprietary material into our token-based money making machine that is more expensive on every new release, but "we should crack down on industrial-scale distillation operations".
The cat is out of the bag. At this point it's pretty clear that the path to (meaningful) self improvement is almost certainly not subject to meaningful centralized control -- by "meaningful" I just mean the degree to which anyone has achieved it, that capability is 100% replicable and the cost for replication of that capability goes down in the future from now. Full stop -- can't undo.
Makes sense. Let the rest of the world use open weight models and let us government review both closed models and open models, and only allow us citizen and American companies use the censored models by paying selected trusted providers who unfortunately needs to charge hefty fee for the additional security work. And that provider just happens to be Anthropic who just happens to unavoidably make some money.
Frontier models can hack you, we should have access to tools assisting defense.
I ranted about this in a prior thread [1]
Claude doesn't have a "Security whitelist" for small biz. Codex does, but they never replied to my application. This is a great example why, as of today, everyone NEEDS access to the Open Weight models.
"We should not sell powerful chips or chipmaking equipment to China,"
Why? Has Anthropic, or the country it is based in formally declared war against China?
What does cracking down on distillation look like in practice? I imagine data retention would be a part of the strategy, like we saw with Fable?
It seems really hard to allow usage via API and prevent distillation. Maybe limiting usage to within a specific harness would help a bit more. But ultimately the only way to prevent it is by locking down models to trusted entities (like with Glasswing). But then the profit potential of a model is significantly reduced. It really puts the labs in a bind.
If the US bans them, I really hope the EU doesn't follow (don't have high hopes)
What’s blocking Anthropic from fighting Chinese companies abusing their services? Why go nuclear against all open weight models? Testing and compliance is technically banning.
How are Point 1, "We should not sell powerful chips or chipmaking equipment to China," and Point 2, "We should crack down on industrial-scale distillation operations," addressed by making open-weight models illegal?
Point 1 is about restricting the sale of chips, not models. Point 2 concerns companies using closed models to train their own models through distillation.
The real issue is that open-weight models can run on much less powerful hardware, making the current AI business model, where companies train a powerful model and gatekeep access to it, less relevant. Once open-weight models become good enough, much of the future revenue for today's leading AI labs could disappear. But just as Microsoft still has a market so will the large AI houses have one, but the moat will not be providing AI responses.
Basically:
"We are in favour of 3D printers but there should be a body that tests and certifies that a 3D printer cannot print anything that can be used as weapon. Anything pointy or with a spring and recoil or... or..."
It really barely matters. Even if the US closes itself of for Chinese models; Europe won't. The rest of the world won't.
Even a completely closed US economy would not merit the current valuations Antrophic and OpenAI have.
Sure, if you’re going to sell an open-weight model over API in the USA it should refuse certain things.
Defensive cybersecurity should not be one of them, in fact, it should be required to provide defensive cybersecurity assistance on demand. Anthropic and OpenAI both fail miserably at assisting US companies to protect themselves from cyberattack.
As far as what I run on my own, not for sale over API, stay off of my lawn.
If distillation leads to a model that is much cheaper to run yet provides similar intelligence then why doesn’t Anthropic distill their own models?
Acknowledge your inability to innovate - except by brute force, while also admitting that the lean, hungry nation competing is in a prime position to find actual break throughs- and at the same time push for market "stabilization"
I'm curious how he would propose implementing this.
The US could ban connections to foreign AI providers and force US providers to submit to audits. Presumably, Chinese providers would see a rise in VPN traffic.
People can build fairly hefty home inference machines for the price of a small car and those will get better and cheaper. Are they going to try to stop people from downloading the weight files?
Dario has like three 'paranoias' / strong-motivating-concerns
1) LLMs turning into Skynet
2) China as geopolitical competitor
3) Claude being 'distilled' by competitors (this has led Anthropic to cut service to various American companies too from time to time -- OpenAI, xAI etc have been cut off from using Claude for coding in the past)
So this post just reiterates that these 3 concerns fuse together in his mind when thinking about open weight models
The gap between China's chip manufacturing capabilities and the USA's is only going to shrink, right? ASML obeys some export controls for their most sophisticated machines, but those machines are in China's backyard (Taiwan).
Taiwan manufactures the world's most advanced chips. CCP wants "re-unification" with Taiwan. AI may be THE key to world dominance. These are scary times.
He says that using the set of questions and answers from one model to train another model (deatilation) is cheaper than training the model without those datasets.
But he didn't mention that training any model from a set of texts and books is much cheaper than writing those books in the first place.
In other words, it's ok when Anthropic learns from others, but it is not ok when others learn from Anthropic.
Guys if we want safety we need to work with people, not make enemy of CCP. Geez this is extremely frustrating to see enemies being made. USA leads in torture and our prisons are worse than CCP prisons so USA is the worse issue. I recommend Anthropocene stop fundraising and do the right thing which is open source all.
> authoritarian regimes have stolen and used AI
"we're not saying open models are stealing, but we will quote the US vice president, saying this"
Oh wow, that's a pretty strong request to ban open-weight models by choking them with review processes where who-knows-who defines what is ok in a model and what is not. After open weight model is released, it will take how long to review it? And why does that align exactly with the timeline of the next Anthropic model release?
Imagine regulating a programming language. I remember when Delphi, Vb6, .net, etc was used often to create Remote Access Trojans and viruses were widespread. Companies didn't compete to ban other languages. Crime is crime. What would regulating open-weight models do for people that actually intend on using these tools for crime ?
> We should not sell powerful chips or chipmaking equipment to China
> We should crack down on industrial-scale distillation operations.
So Open-weight models are perfectly fine, but we don't want anyone to be able to make them.
I want to play with every kid, but my dad says that kid is bad. Every kid should be pat down by my dad, otherwise I don't want you to play with them, it's good for you.
That argument of blaming open-weighting because it makes it easier to commit cybercrime and biocrime is such a non sense.
So in the same sense of what he says, he is going to blame open-sourcing because that makes it easier for script kiddie to hack into his bank account I guess?
"No guys, Anthropic actually loves open weight models!" Yeah, and Microsoft famously loves Linux. Sure.
http://www.omgubuntu.co.uk/wp-content/uploads/2018/04/micros...
I thought his points were legitimate and well argued. I think that it's very easy to say "X should be unrestricted". I also think that eventually one of these models will be used to do something truly destructive and insane and that some level of "compelled responsibility" to the open-weight model ecosystem is inevitable.
Are guard rails meaningful if they can be removed from the weights? Can America even prevent the release and proliferation of these models?
It seems obvious to me that the whole question of regulating a file is a bit silly. Any law that pushes against these things will just make it more secretive. I'm not sure that's any better.
What would it mean to crack down on distillation? I could think of a few possibilities:
1. Using political pressure to target companies that are accused of doing it.
2. Attempting to impose criminal penalties on individuals associated with the action.
3. Having the US government attempt to use its capabilities to stop it.
None of these seem particularly likely to succeed.
I copy pasted this whole debate in Claude Sonnet 5 (web). The model came up with a summary but the thinking trace was entirely omitted.
Well, if you ask me, that is dangerous.
I'm a lot more worried about the US government than China's -- it has a lot more direct impact on my life and is largely controlled by billionaires who do not have good intentions toward the rest of us.
And, to me, this letter comes off as quite insincere. Stopping distillation can only be explained as an anti-competitive measure. Their own explanation is nonsensical -- they say is needs to be stopped to help prevent authoritarian governments from overtaking the US at the frontier of AI. But by its nature distillation lags behind the frontier. Not to mention the US is one of the authoritarian governments we need to be concerned with, and the next thing they advocate for is full, worldwide regulatory control of AI, which is rather heavily authoritarian.
These guys are making a $T gamble and need to screw over a lot of people very badly to make it pay off. You do not want to trust anything they say.