alt Hacker News"he warned it could trigger a large-scale incident this year or next, such as a temporary blackout of financial or power systems."
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.
Replies
I agree with your premise, but not your conclusion. The problem is not the scope or scale of things, but treating them in an absurdly stupid fashion. Why are power systems internet connected? It certainly enables some greater degree of power and flexibility, but it exposes them to potentially catastrophic outcomes in case of malfeasance. So what happened at some point is somebody convinced somebody else that the security was super duper ultra good, and that's simply not true.
And I'd go one step further to say that it probably can not be true. This has always been true and probably will always be true. All LLMs are doing is demonstrating that. And the LLMs of tomorrow will demonstrate that the most LLM hardened system of today is still just a vulnerable mess. So the conclusion is simple - things should not be connected to the internet unless the worst case scenario from hacking/exploiting is relatively low. If somebody wants to connect their coffee maker to the internet, for one can only imagine whatever reason, then the worst case scenario is a broken coffee pot - who cares, go for it. But connecting e.g. power, water, or other core infrastructure to the internet? No, never.
Just because something hasn't been proven to be a problem today does not mean it won't be a problem tomorrow. We shouldn't learned that yesterday, hopefully we can learn it today.
We are starting to get a taste. Credit Mutuel Arkea just admitted having been hacked. Over 100 millions gone.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
You’re describing a weapon, and if AI managed to do that (I believe we are not far away), then this would probably increase the investment not decrease it.
Don't ignore the very real human cost this would bring. Every day the power is out, more vulnerable people would be harmed(including death) directly from not having electricity for essentials or from systemic failures like in food distribution and storage. Financial systems being down could also have extreme impacts on some people, but maybe easier to mitigate via human compassion.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
I agree that people only really respond when something goes wrong badly enough that its impossible to ignore.
I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
To me this sounds a bit like "a zombie apocalypse wouldn't be such a bad thing, because natural evolution would lead to humans becoming faster runners" line of reasoning.