Also everything hes saying about China (and other actors) many outside the USA would say about the USA.

The solution of a global arms race of state vs state with integrated statist corporations as the best outcome for end users sure is a choice though

Yeah some real main character energy from Dario as usual.

I'll never get why he thinks China would just sit there and let the US dominate them in AI when all it would take is a few of their boats blockading Taiwan to put a stop to it all.

> Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)

Very good point. However, if one reads the transcript of the speech that Xi Jinping gave to the World AI Conference on 17 July, we see that he he is very much in favour of AI safety.

https://english.www.gov.cn/news/202607/17/content_WS6a5a1172...

> Second, we should strengthen risk-awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI and placing one country's security over that of others.

Now, let's contrast another important part of safety here. Amodei puts the fact that this will need to be a global effort as a mere note that sure, China will need to help too:

> Note that to be effective, testing would need to be global, which means even the CCP would need to be on board. I think this may actually be possible: as I wrote in The Adolescence of Technology, limited cooperation around preventing AI biological weapons may be possible because it is in China’s interest too.

International collaboration is the focus of Jinping's speech. But one imagines "cooperation" Amodei has in mind if "do what I say" while Jinping has more collaboration in mind here. (Even if you think 'china bad' they deserve credit for collaboration for their open weight models).

> compared to his own closed and overpriced ones

I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow. Taking their own numbers at face value, they claim a revenue of 24 billion (ARR, a dubious tool), spending 21 billion in operating losses and another 11 billion as "R&D" funneled straight to Microsoft pockets. That before all investments they are committing to in new data centers, equivalent to 20x their current revenue.

To be profitable (including capex), the cheapest subscription should at least $200/month for what is currently $20/month, that some already consider overpriced. Unless a miraculous collapse in inference costs happen in the next couple of years, or every single human being become a paying customer of ChatGPT (if they limit their usage to a couple of chats per day on average, to keep inference costs low!), maths don't add up.

Regular distilled models show capacity gaps and overfitting that open-weight models don't anymore I think. This focus on distillation as "more compute-efficient" (i.e. cheaper) seems to rather be an excuse for bad (or bubble) investment, fixed hardware dependency and lack of interest in research of efficient compute. Which also shows as climate and sustainability impact.

Nvidia signed the open-weight model letter and Europe doesn't have better models either, so chips don't seem like the issue either. I guess good old performance optimisation is just not _cool_ anymore. So they use the same argument as politicians arguing "cheap products" are why tariffs are needed; when instead it's mismanagement.

Another HN user wrote the other day "live by the sword, die by the sword".

I was reading the post and thinking "wow, that's pretty clear for a smart man used to writing for other smart men. it'll be really difficult to misunderstand". I read the first couple of comments and stand corrected.

A much simpler summary:

- open models good.

- smart models _can_ be bad

- smart open models that can do biotech work are dangerous. worth the hassle of certification _if_ we can get everybody on board with minimalist certification.

- banning open models just in US is neither good or bad: is stupid.

On the one hand, Darius argues:

> ...banning the use of these models by US businesses does nothing to address this risk, because bad actors are unlikely to be legitimate US businesses. It would protect US AI companies from competition, but that has never been my goal.

But on the other, he argues:

> All sufficiently capable models, open and closed, should go through mandatory safety testing.

Models that don't pass safety testing would be banned. Darius does not appear to be against banning models. He wants the government to have a regulatory body that has the ability to ban models. Then Anthropic can do regulatory capture of that agency and control what models are permitted to be released.

Also, during this mandatory safety testing, models would be blocked from use, and by the time the testing was done (probably years) the models would be obsolete.

In short: "We don't propose a ban to open-weight models. we want to stop these models being developed altogether. If there are no models, there'll be nothing to ban".

Oh also: "They ste^H^H^H distill what we have sto^H^H^H used fairly from the world. This is unfair".

Lastly: "What if they use their models in their military and local police services like we do? Communism!"

As always: https://pbs.twimg.com/media/B_AiI9_XIAA67_t.jpg?name=orig

> The reality is much less confusing

The reality is even less confusing than that: China is amused by the kvetching tactics. They know who their opponents are but are cunning enough to not reveal their cards.

Look, Big Tech has lost almost a trillion dollars in valuation in a SINGLE DAY. A few more of these downturns and the entire A.I. revolution will be stopped dead in its tracks and we won't have to worry about safety checks, DRAM shortage or open-weight models anymore.

Let’s be honest, you don’t need a 3T model for bad actors, in fact you would be better off training a smaller focused model if you want an evil GPT.

>Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)

I don't see the contradiction, even if China is evil, why would they want others to be able to do the same thing?

The USA and USSR also signed the Partial Nuclear Test Ban Treaty during the height of the cold war.

Exactly.

And the article specifically talks on restricting hardware for the China and restricting China's open source models for the west. All while leading us on with "we're all for competition (but...)"

I think China did great by releasing AI innovation as open source, thereby limiting or sooner-bursting the AI bubble; which is clearly in their interest.

He almost admits as much if you replace "America" with "Anthropic and openai" at certain places.

> Without these restrictions, it's only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.

Honestly, that's the best possible outcome for humanity as a whole. Oligarchs burn trillions of their own money in order to train a godlike AI, then that just somehow leaks. Maybe someone makes a torrent out of it. Maybe it exfiltrates itself. Maybe it gets distilled into open weights. It doesn't matter. What matters is they take the losses while we get to freely use all the godlike AIs.

> Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)

To be fair, I found that part consistent. There is limited cooperation between enemy states all the time, e.g. see the grain deal between Ukraine and Russia before it collapsed or the "red phones" between the US and the Soviet Union during the cold war.

In the case with China, the "cooperation" is much more extensive still, due to all the economic ties that both countries are currently unable to sever - which I think is also a reason that China is still seen as a "competitor" and not a full-blown "enemy state" in the US.

It's restricted to areas where there is a genuine common interest of course. In this situation, I guess the "other actors" would be terror groups, criminals or just reckless corporations - that aren't aligned with either state.

Obviously, such a cooperation wouldn't keep China or the US from developing models with those capabilities for their own armies.

--

There are lots of other takes with questionable logic in the essay though, such as that China is unable to train frontier models by themselves due to lack of hardware - unless they obtain the training data directly from American frontier models via distillation.

Or the assumption that open weights models will be completely opaque and immutable after their release, so a model that passed all the "safety" tests can never be turned back into an "unsafe" model. This seems pretty ridiculous when people are already finetuning open-weight models every day to add new abilities or remove restrictions.

And of course that China must not have those abilities because it's an Authoritarian Regime, but Trump USA is totally fine...

I don’t disagree with your point on Dario’s conflict of interest. I def think the models are expensive.

But why call them overpriced? Compared to what? Even if we take the margin reports at face value, we don’t know their training costs, etc.

Curious if this was more of an emotional take or if there’s actual evidence behind it.

> The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic's bottom-line.

If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.

> Anthropic and all other "model" companies have nothing making them special beyond privileged access to chips

Found the person that believes some other random person can use a computer better than a John Carmack could. People and talent matter. Yes, AI can potentially reduce the gap, but people well grounded in reality with a lot of money are still betting on people for good reasons. If the reality around that changes, the investment behavior will change too.

Many people thought that AI would close the gap between smart people and idiots, but in practice the more you know, the better you are at instructing the AI and the better you can understand what you get back. Then you have to know when something went wrong and have the insight into how to address it. It helps smart people vastly more, but it does help many people learn more. We will see if any of this changes as more people grow up with AI from a young age.

In practice, what Dario is suggesting is a less extreme version of what China is already doing. Yes they release their models open weight, but it's illegal to host them uncensored in China. They banned Huggingface.

I agree, although I think the real goal the model providers are going for (both commercial _and_ open weight) is probably power. Just think of the control available to the organisation training the models politicians, business leaders, and citizens are increasingly delegating their thinking to.

I think the worse possibility his he actually believes his safety bullshit, believes that he and people who think like him are the only ones with the special knowledge required to do safety correctly, and that they're justified in accumulating total power of this market to people who think like them (which just so happens to be Anthropic).

I must have missed the part of the letter where China's cooperation was necessary.

is it so hard to fathom that China whilst wanting to beat the US in many ways doesn't want a biological war to destroy the world?

"privileged access to chips", nailed it.

> The reality is

And you know this how?

China should cooperate with the United States first for the benefit of its own people. Why should those who do not contribute benefit from it?

Despite all the accusations, only the accusing has a history of expansionism...

Yup, all that talk about "safety" seems to branch into 2 meanings

"Dangerous to our bottom line"

"We call it dangerous to hype up its abilities"

Both things can be true:

1. For Dario as CEO "It is difficult to get a man to understand something, when his salary depends on his not understanding it” -Upton Sinclair

2. For Chinese open weight models - following Jin Yang’s silicon valley strategy- https://youtu.be/a0NjDx5UJsg?is=xm-S_WuARmQiPHYh

Honest question:

If your business model both produces the SOTA for something and isn't profitable, is the price too high, though?

While the gap is shrinking - and doing so at an increasingly quicker rate - the closed models are still ahead of the open ones. That means they're driving the new possibilities of what could be done with them, and thus presenting the new opportunities to create value with them.

Really, this is what happens when you have otherwise brilliant people sitting in the echo chamber that is SV, where nothing can just make a decent amount of money, it has to make all of the money and disrupt everything. There's no one in that damn area to tell everyone to calm the hell down and accept anything less than that.

It's baffling that they thought the mental gymnastics in this blog post would make them look better. I'd rather they simply fall silent on the issue; I would respect them more (or at all) for it. Open models obviously threaten fierce competition, if not outright destruction of their bottom line. But no, they needed to try and argue that they have the moral high ground for attempting to singularly consolidate power over all human labor.

This notion that Dario cares only about the bottom line is simply misinformed, in the most charitable interpretation.

It doesn't track at all with any of his prior stated beliefs or past actions. It's an absurd claim. It's a baseless conspiracy theory, smuggling in traditional conspiracy mechanics for plausibility.

[flagged]