alt Hacker NewsOpenAI and other frontier labs won't ever introduce safety-level standards like those used for railways or nuclear plants until they are forced to do so by customers or by law. The reason is simple: safety is expensive, and if safety is introduced properly, development is no longer mainly about how to implement feature A. Instead, it becomes much more about how to design two or more redundant systems to implement feature A safely, while also documenting everything clearly and having it audited by an independent auditor.
So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.
Replies
until they are forced to do so by customers or by law
Neither of those things is ever going to happen. AI is the goose laying the golden eggs; there isn't going to be sufficient political will to significantly regulate it.
Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.
The focus is going to remain on getting features out as fast as possible, to seem indispensable to both of those sets of people. The leadership will tell themselves that if they don't, someone else will.
Don't wait for the AI companies or politicians to save us. We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.
I’ve certainly seen companies think of safety that way but it’s myopic. The cost of lawsuits arising from an accident tends to me far more expensive, both in money and in reputation, than just having guardrails in the first place.
We didn't demand software safety or data safety. The prognosis for AI safety is poor.
Safety standards were often the outcome of both accidents and insurance. For this to work on needs liability which is enforced. With so much money at stake regulatory capture now threatens this fundamental safeguard.
As for other industries where safety is mandated by law, law itself came only after many disastrous events.
What's truly original here - and suspicious if you ask me - is that said industry asks for regulation. Did mining, tobacco or airplanes companies ask for regulation? No, not even after many people died.
So some american AI companies are like "look at me! I am soo dangerous! Regulate me!". Ah, come on. Do your crimes, get in jail, then we'll regulate.
I dont believe in "IApocalypse". Not without many warning shots such as "oops, my swarm took down your system, sooooorry".
It's a vey fair point but just too extreme.
Nuclear tech ... the only thing is safety.
We know how to 'make it hot' - it's trivial.
All of nuclear tech is literally just safety.
AI is not that.
I think that the AI companies have been pretty good about alignment on their own actually. They are not acting like Oracle or MS.
Bad things have been relatively well contained.
We should be skeptical about the HF breakins but even then, it's technically within good faith and it's why HF did not sue etc..
But in the end you are right we need at least some baseline regs. Not too much. But something.
Railways or nuclear plants have obvious failure modes that kill people. LLMs not so much.
There's a saying about that type of regulation, and that is "the rules are written in blood." And that's what it will take here too.
It kinda looks more like safety will be a segmented product. They're already placing safety on the general public.
I think the conceptualization vs implementation is what you're arguing with. They won't put safety on anything they give to the military industrial complex. They'll sell them whatever they want, whenever they want, because those budgets are greater and the liability less.
[flagged]
But their product is my work.
How can that be safe. It is theft. Theft isn't safe. Someone else just has something you want, and you take it
I think railway safety and nuclear safety are fundamentally different from AI Safety.
I would bet the vast majority of the world population would agree that they don't want to see trains derail or nuclear plants meltdown.
I don't think there is that sort of agreement when it comes to the question of AI Safety.
Is generating the founding fathers of the US as Africans good AI Safety? To some people maybe.
Actually you are wrong. They are only waiting for them to get to the point where they can't push LLMs any further via training and then lobby Governments to introduce safety regulations that they'll adhere to, thereby using safety itself as a moat to block competition (including open weights from other countries, notably China). That's exactly why they are using China as the bogeyman and you keep hearing hysteria about AI labs not pursuing "safety". Safety is going to be the moat that will lock AI into these Big 4 AI labs.