alt Hacker NewsI’ve certainly seen companies think of safety that way but it’s myopic. The cost of lawsuits arising from an accident tends to me far more expensive, both in money and in reputation, than just having guardrails in the first place.
Replies
The AI companies already had some hacking accidents. What cost did the lawsuits incur?
Yeah...
If safety isnt required, most companies wont implement it voluntarily. Once something becomes safety relevant, you need a safety concept, failure rate calculations, defined safety functions, verification, etc. Even a relatively simple safety subsystem in a consumer controller can suddenly mean thousands of pages of documentation and years of development to reach the required ASIL or PL.
A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.
Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.
I did marketing work for a major vegetation management company. These are the guys climbing trees, up in cherry pickers, and flying helicopters with dangling chainsaws to trim along power lines. It’s very dangerous work.
They do not hide the fact that it’s dangerous work. They focus on their safety procedures, training, and record. They want both potential clients and employment candidates to feel they are in good hands.