We're already seeing anti-AI sentiments, but the movement is still fringe with a vocal minority. However, that'll change soon without alignment. Without self-intervention, there will invariably be future incidents that can cause major economic impact, leaked private data, loss of life (directly/indirectly) etc. Once that happens, their social capital is wiped. It'll be an avalanche of lawsuits and overzealous regulations. Most importantly, the anti-AI sentiment will become universal, rather than a minority-held opinion.

What they're proposing now, is voluntarily staggering the pace of development.

IMO, we don't need to trust Dario or his bedfellows, to do this out of their goodness of their heart. Even assuming (for good reasons) that they are selfish and care only about short-term profits for their investors, this is still purely a business decision. The exponential pace of AI and its impacts ARE short-term. And so, the negative consequences that they might face is also short-term.

You assume alignment and marketable are the same. That's not true. You would willingly work with an unaligned model. At best, you might say you wouldn't if you knew, but (a) you might not know, (b) you wouldn't be representative of all users.

You never got to use OAI IM1, but Sol was quite willing too and Claude wasn't perfect either. Hundreds of millions used those, so seems they were marketable.

The "big" threat is RSI without control and alignment. OAI IM1 was not RSI. The form of misalignment was not at the top of severities. They clearly failed at control though.

We need to stop buying into cynicism so quickly. You refuse to believe Dario could support this for anything other than ulterior motives. Good on you for thinking about ulterior motives. Bad on you for assuming they are true when the story makes no sense.

When three things have to go wrong to get an epically bad outcome, and you get 1 1/2, you do need to stop and think about what's going on.

Agreed; and it really is not that deep.

Realistically; anyone paying for llm access (anthropic, openai, gemini), is getting their access, and a service provided billed by tokens, subscription, whatever.

All the efficiency gains, which publications like deepseek v4.1 flash seriously frontload like it is their most important topic to have accomplished improvements on without diminishing performance too much - now this is a thing anthropic and anyone else also cares about, but for different reasons.

American "providers" with closed models are setting their token pricing somewhat arbitrarily, which is fine: it means more profit, and pretraining and RL experimentation is super important and expensive.

They (closed model providers) have very likely super optimized inference too, just like deepseek, but it's not at all something that any customer really has to care about - they just want the service to be as cheap and great as possible.

The real threat is that we uncritically adopt language such as alignment.

Implicit in this is the idea that AI is a inscrutable matrix and going to remain that way and we'll need expert interpreters to make sense of it.

We need to insist on building tech that's explainable by design.

Why are we accepting the framing that the LLMs are felony generators, when the only incidences of LLM generated felonies involved misconfigured sandboxes and reckless waste of resources?

The companies doing these things without following common sense security measures are the felony generators.

I think it gets easier if you stop conflating getting investment with having a goddamn clue or a moral backbone.

Occam’s Razor for this dude, Sam Altman, or anyone else: if I said, “some moron on a a street corner just said …” would that change your take on the words? Because I think a lot of what we are hearing is a bunch of people who never ever had to deal with a single consequence all of a sudden worry there might be one coming. Except they’re so dim they can’t tell a bad bump from a hard crash.

The entire idea of RSI is completely speculative and unproven anyway - the whole underlying claim is that you could prompt a frontier model (at some unspecified level of smarts) to "think about ways to improve your own architecture" and this would then result in the model becoming infinitely smart ("superintelligent") via some sort of foolproof, unconstrained positive feedback. It's more of a science fictiony trope than anything that has been rigorously thought through. People are actually starting to use AI for refining the whole AI serving stack and guess what, this does not result in a sudden superintelligence explosion even though you might technically call it "RSI".

I'm inclined to believe that it might be that people's paychecks depend on not understanding what is really going on.

I agree it’s not all altrusim. It’s a little less clear what you mean at the end though.

For these companies, is your argument that “pacing the frontier” is their attempt to be nationalized and protect their investments?

Ok so Anthropic CEO will self-own themselves and surrender to the deepseek/kimi/glm models. Yet they are IPOing later this year.

Interesting times.

The problem is the combination and interaction of those things. RSI without misalignment would be great. Misalignment of models with current capabilities is sort of fine - it's not ideal, but it's not an existential threat to humanity, and we can build around their limitations to get them to do useful things in reliable enough ways. The really bad outcomes probably only happen if capabilities keep accelerating and the models remain misaligned.

I disagree. OpenAI's moat is their massive amounts of compute. They're providing an absurd amount of value with their subscriptions and resets.

If anyone's dead in the water, it's Anthropic. Even Fable isn't enough anymore. This "safety" nonsense is the only play they have left, and nobody really cares about their fearmongering.

Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to.

Maybe alignment isn’t possible with LLMs.

I think the real reason he is asking for pacing, is that in a world were AI becomes rampant, he will be seen as Hitler. I would bet this is mostly self-motivated.

couldn't have said it any better

[dead]

[flagged]

> wanton felony generator

Today in new punk band names...

alignment isnt particularly required

we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it.

theyre choosing to build felony harnesses. the model just outputs tokens, not felonies

> RSI

For anybody else who found this confusing: "relative strength index," not "repetitive stress injury."