logoalt Hacker News

zardo • yesterday at 7:57 PM • 1 reply • view on HN

Email going down is the proximate cause. A system configuration relying on a single unreliable communications route is further up the causal chain.


Replies

buran77 • yesterday at 10:56 PM

Let's not forget that we're talking about people getting hurt, that's the core topic here, yes? The most reasonable immediate cause for the people getting hurt is that there was too much chlorine in the water. Because the dosing system pumped too much chlorine AND the monitoring system failed to alert. Because it was set up to only use email AND the email failed. Because someone deployed a wrong config on MS's cloud infra. And so on.

Not one but two of the hotel's systems failed to operate properly. The chlorine pumping system pumped too much chlorine and the last resort alarm failed to alert.

As I said, there's plenty of blame to go around as a matter of general principle. But for what happened here the hotel bears the responsibility towards their guests, not the pumping or monitoring system developers, not Microsoft. The only reason MS in the title is because that failure is big news these days.

On a technical note, even if they had a backup email provider they'd still be screwed because the people getting the alert were probably employees on the single provider (back to SPOF). SMS could have been a good parallel option. Ideally also some physical control panel that triggers an alarm when any safety critical condition occurs. On the chlorine pumping system, that shouldn't be a dumb preprogrammed pump, it should come with it's own sensors and detection loop independent of any external system, something that proactively stops it before it breaches the safety limits.