alt Hacker NewsI doubt AGI will be used for life-or-death systems like that if it resembles anything like the LLMs of today. Systems at that scale are about risk management, and everything we know about AI suggests that it cannot mitigate risk at all. All AI is seemingly a net-positive risk factor.
Really, the "big AI story" will be when the next Snowden defects from the NRO and tells us what SENTIENT is doing. Until then it's just going to be a bunch of pie-in-the-sky wishcasting about what we want the future to look like. It's getting a little tiring hearing people insist that abundance is around the corner while surveillance is here-and-now.
Replies
The way I read it was that he expects large industrial systems with AI to have safeguards to "fail closed."
Which: it seems like we're already getting to the point of capability vs. unexpected behavior that having the ability to "trip" a complex system may be an important design criterion.
I mean, there's plenty of harm these systems can do besides trying to turn us all into paperclips.
That assumes people mistrust LLM output, which is a perspective inside the HN bubble.
Outside, almost everyone I know - including highly educated people - takes even Google's search LLM output as fact, without a second thought, and even regarding serious issues (e.g., medical problems). Nobody raises a doubt or an eyebrow. A recent article says members of US Congress use it extensively for serious work. Lawyers use it to file cases. Doctors use it to summarize essential medical information. I even hear that software developers use it to create high-risk, high-value code.
In the public people think computers are superior to humans - more authoritative, objective, perfectly logical and error-proof. SV pushes that narrative, for example saying computer-driven cars are safer because computers don't have human flaws (that old sales technique - compare only the competition's unique flaws and don't mention your own; computers have their own flaws that humans lack). And of course, that one is popular on HN.
(IMHO the way to educate the public is to spread the narrative that every computer is created (designed, programmed) by normal, flawed, biased, etc. humans, and controlled by other flawed humans for their own agenda. And that computers have their own flaws.)
I’m absolutely willing to believe there are some people that stupid.
I doubt anyone who works on reactors is that stupid.
But some idiot somewhere is going to put some industrial stuff under AI control pretty soon and murder a few people at minimum.