How Complex Systems Fail (1998)

I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing.

The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure state. Naturally, the "root cause" seems like lock system resiliency. But definitionally a metastable failure is one that persists after the inciting condition is resolved. Now you have two "root causes", the lock failure and the metastability of the deployment system fault. Keep looking and you'll find more.

But to me the biggest brick to the forehead in this piece is further observation that random things are failing all the time in any complex system. "Complex systems run in degraded mode". Resilient components are good, but it's the resiliency of the overall process that orchestrates the whole system that determines whether things are going to blow up.

All practitioner actions are gambles. I should have that inked somewhere.

"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously."

This very much resembles Admiral Cloudberg's write-up of the National Airport collision:

"No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost."

https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...

> Failure free operations require experience with failure.

This is why we created Chaos Engineering. By constantly forcing failure, it made us always create systems in defense of that failure, and gave us great data on where the tipping point is for different systems within a particular failure mode.

I may have shared this before on a different submission: John Gall’s books are really good on this topic: General Systemantics [https://en.wikipedia.org/wiki/Systemantics]

Whenever I see this [excellent] paper, I am always struck by the first sentence, in the first section:

> All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by THE own nature.

(Emphasis mine)

I am not sure if that's a typo, or some writing convention that my uneducated mind can't grok.

This is very important work. If it interests you, and you want to dig further, it's broadly called "Safety II". You probably want to read Erik Hollnagel [1] or Sydney Dekker. I put together a presentation once about how this translates to IT security [2]. But broadly:

Safety I Learn from Errors, Safety defined by absence,Reactive approach, Understanding what goes wrong, Accident causation models, Avoidance of errors, Reducing losses

Safety II Learn from successes, Safety defined by presence, Proactive approach, Understanding what goes right, Repeat what goes right, Enforce successful behaviours, Create new processes based on successful behaviour

If anything, think long and hard about this: "Post-accident attribution to a ‘root cause’ is fundamentally wrong." It's a true statement. If you truly understand this sentence, you have mastered Safety II to the level that a light bulb will come on, and you won't be able to un-see it. This light bulb is a curse, a beautiful curse.

[1] https://safety4sea.com/cm-safety-i-vs-safety-ii-an-overview [2] https://www.msoos.org/largefiles/safety2.pdf

The definitive work on this topic is Normal Accidents, with a modern retelling in Meltdown.

https://en.wikipedia.org/wiki/Normal_Accidents

https://en.wikipedia.org/wiki/Meltdown_(Clearfield_and_Tilcs...

Highly recommend "The Logic of Failure" by Dietrich Dörner (1997)

Worth reading Leveson’s works on safety engineering.

I think there are a few common themes to the failure reasons, but cannot get my hands on it.

This seems like a list of reasons while I am looking for more abstract directions on how to prevent them.

---

I am trying not to use AIs to just do that for me to tinkle my neurons.

Overall a good collection on the complex systems, somehow it missed the part how complex system came into the existence in the first place itself.

> Human expertise in complex systems is constantly changing

I feel this is single most important factor responsible to both making system complex and at the same time improving them depending on who the people are and how they take failures and breakdowns. I find it funny but we are in an era where folks building ML systems do not seem to remember the direction in which to open the screws. They do have clear expertise in something new but clearly lack in some other areas

All of this sounds just like any air crash investigation I ever read

I can't tell if the article is describing how complex systems fail or if they are using failure characteristics to define complex systems.

One of the great documents of our civilisation

Good take, no notes

[dead]

[flagged]