logoalt Hacker News

Software is breaking down at a systemic level

330 points • by zero_shift • yesterday at 6:17 PM • 209 comments • view on HN

Comments

doug_durham • yesterday at 6:39 PM

This narrative isn't consistent with reality. Cloud computing is much older than 10 years ago. We were doing production grade cloud computing over 16 years ago. The OSS revolution dates to the late 90's. The author seems to think that SWE was in a golden age 10 years ago. Sorry, but it's about the same. We have the same problems in SWE as we had 40 years ago. The scale and ambition of the problems we are solving have increased as compute capabilities have increased.

➕ show 10 replies
CharlieDigital • yesterday at 6:39 PM

It doesn't have to be this way.

AI is great at following patterns. AI is great at writing tests. AI is great at doing work that humans would consider drudgery and then applying "enough" judgement to it.

The problem is that most software teams and most engineers are very, very bad at the thing that AI needs to succeed: technical writing. Documenting the patterns, the rules, the heuristics, the SOP, the judgement; developers could barely be bothered to write comments before (which in the AI age are free, infrastructure-less memory that is always "line-of-sight" for the agent and can be used to enrich with keywords).

It's not that AI is the problem, it's that we humans are good at resolving these constraints in the real world at a human pace. AI makes that pace impossible for a human to keep up as a trailing activity. Rather teams need to invest more in technical writing and producing the corpus that AI relies on to learn the operational constraints and rules.

➕ show 14 replies
illuminator83 • yesterday at 10:12 PM

Software is not really being more broken today than it was 10 or 20 years ago. It actually has gotten better. I've seen the codebase of a dozen companies and worked with hundreds of peers. There is plenty of horrific bugs and bad software outside. The reasons are plentiful too and include everything like budget constraints, management failures and engineering skill issues.

Today, the average code that even a product manager or junior engineer submits is actually excellent overall - if your processes, harness and CI etc are well designed. It is actually super easy to author software well now. Whether you build something that is actually needed and is tasteful is another matter.

99% of the security vulnerabilities being found and exploited today were created by people. Cutting most of them out of the technical details is actually good.

➕ show 1 reply
rglover • yesterday at 7:33 PM

Imo: the way we make software is breaking down at a systemic level.

The harsh reality is that some people/systems are adapting just fine to the new way of doing things, others are making a mess of things by rushing via vibe coding and not checking the work (either by choice or by circumstance at work). The results won't be (and shouldn't be expected to be) linear.

The 2010s brought on a very groupthinky dynamic to software dev. This led to a culture where the majority of developers just ask "what is everyone else doing." Now that we're past the blog era of development, that shared reality appears to be breaking down. Now the answer really is "whatever makes sense for/to you," but I think (anecdotally) that's freaking a lot of people and teams out.

For the first time in a long time, a "best practice" doesn't really exist.

➕ show 2 replies
ljm • yesterday at 6:45 PM

Software wasn't exactly hard 12 years ago either, the barriers to entry were being lowered at a rapid pace. 2014 was the dawn of the software bootcamp where JS became more and more pronounced in web architecture.

I feel this piece is either more alarmist than it deserves to be, or it is late to the party.

It uses various incidents in 2026 to prove a point while dismissing those that did take place in the decade or so previous. For example, heartbleed, the log4j RCE, Rails' mass assignment DOS, Cloudflare updating a WAF rule and breaking the internet, Atlassian deleting almost 1000 of its customers' accounts and data, the Ashley Madison breach, Crowdstrike bricking their customers computers, Okta getting owned multiple times, etc.

My own point is that software became like this when we started becoming pragmatic. Nothing to do with AI or management really, we just realised, especially with web tech, that you can release something and then fix it. In the old age of getting CDs or floppies printed and distributed, that was not even remotely feasible. A similar incident in the 90s would involve a product recall or a way to get updated media from a retailer with the fix. Very expensive.

Standards have dropped and continue to do so, because we can also fix fast. It's not suddenly breaking down. If anything, it's just too centralised.

➕ show 1 reply
j2kun • yesterday at 8:03 PM

> This changed the balance of power between workers and management.

I think this is the core insight that I keep coming back to: technological advances always entail a transfer of political power.

We would all be better off, when evaluating new technologies and listening to their evangelists, thinking hard about who gains power and who loses power as a result of adopting that technology.

➕ show 1 reply
MiroslavPokorny • yesterday at 11:34 PM

Systems are failing because everything is getting more and more complex with more dependencies on other systems.

The recent shutdown of the US airports is a perfect example of this. The current system is old, with old computers and i believe old analog networks. They worked but one small change installing a routers and some other new systems caused something bad to happen.

In Australia not that long ago, there was a overhead power line that fell and this caused the trains on that line to stop. This then had the ripple effect of causing other lines that feed the initial broken line. Traffic on roads now also drastically became worse because more people were trying to drive or catch buses and all that.

Everything is more complex and connected.

The cloud is a part of the problem, its just another layer that can and does go wrong. Previously when teams deployed to their own servers, they didnt have to also configure the cloud. They were also not a passenger on the nuances and possible failures of the cloud system from its own management technical and otherwise.

LPisGood • yesterday at 6:26 PM

Software has always been broken at a systemic level. It’s a complicated continuously changing web of abstractions and abstraction breaking hacks attempting to survive prolonged contact with reality.

➕ show 5 replies
dividedcomet • yesterday at 6:25 PM

This feels a little hyperbolic, the grounding was a result of a cable swap and the ssh cert incident they cite was so human caused it’s not even funny. I think things are a little more shaky, but not massively so. I had the town I was living in during college lose its internet and phone lines due to a cable being cut on a mountain pass and it was fine. If your business is sensitive to an email outage, prioritize high reliability over integration advantages that outlook offers. The php2shell vuln is bad, but we also had log4j years ago.

➕ show 2 replies
vb-8448 • yesterday at 9:04 PM

IMO control has been lost because we decided that moving fast is the most important thing and outsourcing everything that is not directly related to business (sometimes even business itself...) is a great idea.

The seed was there since at least a decade, AI just accelerated everything.

➕ show 1 reply
teleforce • yesterday at 11:29 PM

From the comments section (richardstevenhack):

>I've said for at least the last twenty five years or more that software engineering...isn't.

>It's not "engineering" at all. It's still a craft.

It's about time software engineering is taught inside Engineering school rather than being outsourced to Computer Science.

anonreplier • yesterday at 6:29 PM

There are some decent points, but this is collating a lot of separate incidents with wildly different causes that just happened to occur in a close timeframe.

➕ show 2 replies
drooby • yesterday at 6:30 PM

Most folks here didn't even know, but a Bitrise security outage hobbled our entire CI this weekend. Friday to Sunday night.

The security issue motivated all Bitrise customers rotating their secrets. this was a big deal for mobile deployments but didn't get much attention on HN..

sdevonoes • yesterday at 8:52 PM

Imho, the problem with AI is that management thinks that engineers can now do 10X (or do one item of work in 1/x)… and so there’s so much pressure to get things done. That’s how (tons of) bad software gets out in production

Imagine using AI but still producing the same amount of output/outcome. That would be delightful, great software could be possible… but it’s not “productive”

➕ show 1 reply
rphv • yesterday at 7:42 PM

This is the usual reaction when people run into an uncomfortable truth. We're all bad at objectivity - we preferentially look for evidence that supports the outcome we want (or the one that's least uncomfortable). In this case it's pretty obvious that AI is better at coding than humans. Lower failure rate per task is better code. We're just shipping more of it. Nobody's claiming AI code is bug-free. Just that it's better and faster than how we used to do it.

➕ show 2 replies
spiralpolitik • yesterday at 6:36 PM

There have been parts of the software industry that have been mission critical from day one and have built processes, languages, and tools to make sure things are 100% reliable. Building software that way is slower and more expensive and in the past has been an overkill for most systems.

The problem software engineers are facing is that now that software is ubiquitous, almost everything needs to be 100% reliable (or close to it). This is coming into conflict with the move fast and break stuff everything needs to be done this quarter at minimal cost mentality.

Hence why we are at the breaking point.

➕ show 1 reply
tflinton • yesterday at 6:43 PM

I’ve had AI running to auto upgrade our infrastructure and it’s successful twice as much as humans. Failure rate of a human managed upgrade is 1/10, failure rate of an AI agent is 1/20 in my small set data.

Problem is I’ve had 5x more incidents. AI is operating at 10x the speed of a human so even though it’s failing less, it’s causing more incidents per month.

I wonder if people are ready for this kind of paradox where using AI may be safer and faster but less reliable for “human time”.

➕ show 1 reply
adverbly • yesterday at 8:11 PM

I agree with the premise that things are breaking down, but I'm not sold yet that the problem is a systemic one.

At least some of these issues are ones that already existed, but are now being discovered by AI.

If that is the case, we would not see the exponential increase that the article mentions, but should instead see something like a temporary blip where discoveries in existing code spike, but eventually all get discovered and fixed.

I would love to see some more data that specifically looks into the number of issues caused by pre-ai code versus post-ai.

That being said, the comments about decoherence strongly resonate with me. Things are moving so fast in so many different directions right now. I don't think we know the right way to build things in this new future yet.

➕ show 1 reply
bob1029 • yesterday at 6:45 PM

It feels like we are currently experiencing a soft version of the sewing machine riot.

Perhaps no one is intentionally destroying anything in this case, but malicious compliance and simple negligence can go a long way when it comes to disruption of complex systems. Destroying a sewing machine requires actual violence. An air traffic control system is an abstract concept. You don't need to touch it to break it.

skeledrew • yesterday at 6:45 PM

Things are breaking down because they were weak to begin with, and demolition is needed in order to rebuild better, stronger. This is just a period of trial by fire: iron has to be heated and beaten numerous times (with carbon) before it becomes steel. All that's happening is AI is accelerating the evolution of industries into something which all eventually have the strength of steel.

Rustam_Khasanov • yesterday at 7:36 PM

I write code with AI daily and it got much faster. If it brakes - it gets fixed then. We face new vulnerabilities, but we find them faster and have better instruments to investigate and fix. The current period is transitional, but for me it's obvious that it AI is a win for software development

➕ show 1 reply
jstimpfle • yesterday at 6:32 PM

My impression is it's always been a mixed bag. The level of bugginess has been relatively constant. In the meantime, scope and capabilities have gone way up. I hate Teams and Outlook and the virus scanner on my company's laptop though. Those are keeping me from doing my work. Windows + anti-virus makes file access and process spawning 10x to 100x slower compared to a Linux machine.

tcoff91 • yesterday at 6:22 PM

There were package managers long before npm. Maven, rubygems, etc...

➕ show 2 replies
keithnz • yesterday at 6:39 PM

To talk abstractly about all software, you'd also need to analyze all the software that stays working. This article is just cherry picking a few recent incidents and try's to make some conclusions without really looking at causes and whether that can be generalized to "Software".

sega_sai • yesterday at 7:28 PM

I think the article has a point that the if you have a very complicated legacy system, then if you just let an LLM run on this, there is a significant chance that some parts of the system will be become inconsistent with each other, and the whole thing will be more of a mess and more likely to break.

In my view, however, what is more likely to happen soon is usage of LLM to transform various parts of code into the form with much more clearly defined interfaces/rules/boundaries that can be essentially proven to work correctly. That in turn (in my view) will enable much more robust systems.

kaboomshebang • yesterday at 7:22 PM

Thanks to the author. It's a good read and interesting HN comments. Nothing wrong with having concerns, and I'm sure many people share those concerns. It's scary sometimes. But, the view of the author is somewhat on the negative side. I feel that AI development is still in early phases and people or companies are discovering what works and what doesn't. AI empowers us in new ways with new problems, and new trade-offs. The old way of doing things had different hurdles. Interesting times. It's not going anyway any time soon I expect ;)

herval • yesterday at 6:27 PM

This reads like nostalgia way more than reality

➕ show 1 reply
agosz • yesterday at 6:57 PM

The author seems to have glossed over the viruses of the early 2000s,or the Y2K panic, and other incidents from before. Building software has always been hard and it's always been abstractions on more abstractions.

The one compelling part is that now we might be forced to pay for some of the rickety towers that we've erected because AI has made exploits pretty cheap to try.

➕ show 1 reply
SoftTalker • yesterday at 6:34 PM

> You can find programmers who know how to manage the complexity, but they want to be paid lots of money.

This wasn't always the case, by the way. It started with the "dot-com" mania and VC investment in the late 1990s. Before that, a programming job paid decently well but it was on the level of other cost-center professional jobs such as accounting. A comfortable living, but nothing extravagant.

➕ show 1 reply
andai • yesterday at 9:30 PM

See also:

Preventing the Collapse of Civilization - Jonathan Blow, 2019

https://www.youtube.com/watch?v=pW-SOdj4Kkk

Which focuses on software, but gives many historic parallels of societies forgetting how to do things.

Previously discussed:

https://news.ycombinator.com/item?id=19945452

https://news.ycombinator.com/item?id=25788317

asdfman123 • yesterday at 7:16 PM

Software isn't breaking down, people just haven't figured out how to use AI yet.

It reminds me when I first started work on a codebase with proper unit tests: I thought I could ship whatever and the tests would catch it. That was incorrect; while unit tests are a guardrail, I still had to properly test my code.

Similarly, people need to learn how to use AI and make sure they don't introduce bugs or slop into the codebase.

But this brings up a larger point for me: most people are grieving the loss of what their jobs used to be, and denial is a core phase of grief.

I went through it too. I tried to find ways to say AI was silly or wouldn't work. But now it's inescapably here and we need to work in the new paradigm. There are plenty of valid criticisms, but it's already very good, it's getting better, and you have to learn how to use it as best you can.

adamddev1 • yesterday at 6:32 PM

People keep believing that the big LLM players with somehow disprove Rice's Theorem. "Somehow we'll be able to get AI to check if a program does X or Y." "We'll be able to have the machines verify things automatically." But this is is of course as hopeless as solving the Halting Problem. Theoretically, we know this is impossible.

Empirically, we can observe the breaking effects of this broken thinking. Code quality goes down, and apps go from being 99% to 95% reliable. But that's not a big deal, they say, that's good enough... But when the quality of all the building blocks further down get degraded in the same way, you can't keep track of which parts are broken, and which parts are dependable, and the whole thing comes crumbling down.

People keep saying, "I can build an app with AI, therefore we can build everything with AI." This article talks about some potential anecdotes that could help to show that doesn't work.

➕ show 3 replies
mindwok • yesterday at 9:46 PM

There's a lot of writing going around that we are all headed for some catastrophe, like everything has been building to this grand moment where society just crumbles and AI is the final piece.

I highly doubt this is going to happen, guys. The world has changed and software is never going to be the same. The same battle against complexity and hard things will continue to happen, but the front has moved.

Open source as we know it is probably dead. Tech companies that would create SaaS for X and get billion dollar valuations are also likely dead. DevOps is dead. Other stuff will take all these things places in time but it's going to take probably a decade. Get used to it and let's get on with it.

travisgriggs • yesterday at 6:36 PM

> About ten years ago, two small revolutions happened in our industry ... And then, we had the Open Source boom.

This seems a bit myopic. Or naive. Open Source "happened" a lot longer than 10 years ago. Microsoft didn't shit itself about Linux just 10 bare years ago. And it's not like prior to 10 years ago, the narrow focus of npm world wasn't cobbling web sites together with bits and pieces garnered from wherever.

That said...

> But AI is both a cause and a symptom

This I agree with 100%. The rise of AI and necessity of AI to continue to make progress is a massive indictment against an industry that prioritized more features and uptime over solid first principles for years, taking out promisory note after note in complexity, in hopes that the stuff papered over would quietly just fade away.

ellyagg • yesterday at 8:33 PM

I don’t need to read these articles, it’s enough to read the headlines. Software isn’t breaking down, it’s accelerating, obviously.

Software was never engineering. It’s more like writing a novel, which is way harder because the success criteria is ambiguous.

Now some folks are finally being forced to understand this reality.

rurban • yesterday at 6:31 PM

He obviously is preparing another doomsday conference talk. Nothing interesting to see here.

On the other hand software is improving dramatically at a systemic level, even without safe system languages arriving on the horizon.

gkcnlr • yesterday at 6:58 PM

> Most SWEs now are unable to tackle engineering problems without some form of AI assistance. That’s not because they’re incompetent - they just can’t keep on top of the complexity. Even if they could once, they now face a tsunami of AI generated code.

That's called outsourcing, which inevitably enables "vibe" effect of the coding: Your AI goes out of its constraints and does totally irrelevant shit you didn't ask for or does something barely passable you could approve, without knowing further consequences because you outsourced your thinking and control over the project.

To try to find a remedy for this shortcoming, they invented "skills", which totally negates the purpose of an AI (LLMs, genAI, whatever you call it to describe the current tech) IMHO.

It's also funny reading these comments. Most say "well, it was already broken" and they continue trusting AI more. Well, AI is not exempt from this "brokenness".

tippytippytango • yesterday at 6:47 PM

Is defect rate increasing or is the amount of, and use of, software increasing? What is the defect rate relative to that?

➕ show 1 reply
tracerbulletx • yesterday at 6:34 PM

When I first started my career we used to ship like once a month and that was considered good, everything was extremely convoluted and worse than it is today, whole segments of the industry didn't even use Version Control, half the government websites you used stored passwords in the clear and emailed them to you for resets, e-commerce was a cgi script that sent an email for fulfilment and it took weeks to get your shipment, sites went down at night and weekends for maintenance regularly, servers fell over with traffic that would run on a raspberry pi now, but for some reason everyone keeps saying how bad things are now. They're delusional or don't actually remember the past.

dathinab • yesterday at 8:08 PM

Honestly there are 2 overlapping problems:

1. go fast and break things became the norm not the rare exception, mostly due to under-regulation when it comes to responsibility and other much more complex side effects of what you can describe as the US forgoing a fair free marked in favor of out of control monopoly markets they can use as soft power against other countries

2. software being really hard and most tries to fix that being stuck dead before maturing due to them being unfamiliar and often complex to get started with. There being a huge problem with opposite bad ends of "misguided simplicity" and "programmers loving 'clever' & overengineered" didn't help exactly either.

> It wasn’t always like this.

no it was, yes the 2010 felt like a high time. A lot of 90th nonsense has finally been left behind (but would still lead to security issues years later, e.g. log4shell in 2021); Tools for analysis had gotten better; Compute allowed expansive test suites and "go fast and break things" hadn't yet been spread to widely to critical infrastructure (at least kinda, _a lot of the issues with Azure today root back to decisions done in the 2010th_!!).

Most importantly while having well educated highly skilled Devs is needed, _it isn't sufficient_. They still are human. Blaming the huge flood of CVEs on AI generated code is simple, but wrong. Many of the CVE in recent years are human caused, some older then usable AI code gen and at least in case of the Linux kernel much of the code of the past had been written and reviewed by pretty professional, skilled, well educated developers and still AI found way way to many CVEs in it.

So yeah IMHO there is a long standing fundamental issue with how to handle software production, starting with the very first step of requirements engineering (which often isn't done, or at least not properly) and ending with how we fundamentally compose software components.

And the problem I see with AI it makes it worse. We already where approaching a breaking point with software dev and I hoped that this would create pressure to finally find a working way to reinvent it from scratch. But now AI allows us to apply the same failing approaches even more large scale, faster, cheaper and sure with more bugs, but also with AI making many bug fixes far faster and cheaper (if correctly used, it often isn't).

throwawayffffas • yesterday at 9:23 PM

I mean all of the failures that happened have happened in the exact same way many times before, default passwords not changed keys signed with 5 and 7, we have seen all of this before, and the FAA issue, was it even software, I have not followed the coverage, but its systems setup 20ish years ago and forgotten.

Sure the MS thing might be vibe coded horri, but it's not like we have not seen outages like that in the past.

If anything AI is bringing all of the bad configs and legacy issues to the fore front, because "it" has the time to look for them.

titzer • yesterday at 6:41 PM

Not everyone is using AI to make shittier software. Most big important software projects I know of are frantically fixing an avalanche of AI-found bugs. Important software is getting more robust and resistant to attacks, while still adding functionality.

There's a lot of slop, and it will almost certainly grow quicker than non-slop, but there is a big opportunity for those of us who still work at the bottom to make the pillars more solid, and take on re-engineering projects that would be infeasible otherwise.

Case in point: Before Claude a port of my compiler to a new architecture might take anywhere from months to a year. After AI, now that there are multiple backends as examples and an even larger--massive even--test suite, this can take days or less. It took Claude about 3 days to come up with an arm64 backend for Virgil (around April time frame). Then in August it took just 3 hours to build a complete RiscV backend. Both of them pass the entire test suite. And they shook out other bugs elsewhere. Overall, more targets, and more robust.

Things can have rocket boosters, you just gotta know where to place them.

_pdp_ • yesterday at 6:47 PM

On the flip side, once you hit rock bottom the only way is up.

chalmovsky • yesterday at 6:45 PM

> I believe that tech companies risk losing their “organisational cognition” - their ability to think and respond as a coherent unit.

LMAO. They can't lose something they never had in the first place.

This article has other pearls like this:

> Reddit and Bluesky speculated that even MSFT engineers had no real handle on the complexity of this outage.

Like, with all due respect, what the fuck do they know?

Also this:

>It’s the software that controls your flights, your banking, your petrol pumps, your TV, your payroll.

Dude, how are you gonna sneak TV software into the same sentence as flights and banking? :D

This is not a serious article, and I don't recommend reading it. But to finish on a positive note, since I've already (regretfully) read it, here's a sentence I like:

> This is the real threat: a scenario where nothing in the system is done with intent, and nobody in the system has control. We generate complexity faster than we can tame it.

I would just add that this was the case long before AI. And also, just as AI generates complexity at such a great pace, it can just as easily help you fix it and fight that complexity.

cadamsdotcom • yesterday at 7:11 PM

All software has bugs, and some software is critical

Which means all critical software has bugs!

There's also more of it! Even more once you finish reading this sentence. If security incidents scale sublinearly vs the growth of critical software, we should be proud.

Crappy work is not universal either. There are engineers out there, guiding agents diligently, doing a good job. Never letting an agent go wild unattended, writing the same code they'd have written themselves, but with more tests and more better assurances and more docs and better infra and and and.

The problem that leads to blog posts like this is that good, diligent work, like critical software that does its job, are invisible.

Because no one notices a building that's not on fire. They just walk right by.

acedTrex • yesterday at 7:16 PM

Who could have forseen this except literally anyone with a braincell

afterthunder • yesterday at 8:25 PM

Most people here are dismissing this article just because it doesn't use variations of 'fuck' 50 times or throw in emojis to make it 'fun to read.' And as always, decent comments only appear at the end of the page.

mihaaly • yesterday at 8:30 PM

"Decoherence: the accumulation of locally valid but mutually incompatible or redundant models within a system, such that the cost of change is dominated by translating among those models and preventing unintended side-effects."

This one is interesting. Not on its own merit which is more like a mother of facts, prime phenomenon, but that it was mentioned at AI. I had this feeling of decoherence with all the various techniques and approaches that existed out there, from programming languages through data management through composing instructions through user interactions, into the way orchestration of all of these, way before AI. Somehow none were matching the specific composition of products needs completely so had to tape together available parts at loose couplings, hammering things a bit if needed. Or choosing those that fit together perfectly on the cost of giving up on requirements/functionality for the product. Lots of good libraries not good enough for some details so a new one emerged or the existing one tried to be so many things that it reached digital schizophrenia stemming in complexity (allowing flame or even religious wars between developers about best/sacred approaches/techniques/mentality/...).

With AI I feel different: the promise of purpose made things fitting together well.

🔗 View 18 more comments