logoalt Hacker News

nickcw • today at 11:34 AM • 10 replies • view on HN

From Google's status page https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPy...

    We are experiencing an issue with multiple services in
    europe-west2 beginning on Monday, 2026-10-19 23:08 PDT.
    
    Due to a power event we had suffered an outage within
    europe-west2-b zone. The power event is mitigated. Partial
    mitigation to infrastructure services will be observed within the
    next 30 minutes and the remaining services will be mitigated there
    after.
So some kind of power failure happened to bring down one zone. Unfortunate, but this kind of stuff happens. That is why all the cloud providers have multiple zones, and I'm amazed the big companies reported as down weren't using multiple zones.

(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)


Replies

bsuvc • today at 11:55 AM

I'm not saying it's the case here, but sometimes the cloud providers themselves have inter-zone dependencies, or even inter-region dependencies that can cause outages in other zones and regions.

For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...

➕ show 3 replies
robinson-wall • today at 11:49 AM

> I'm amazed the big companies reported as down weren't using multiple zones

Much as with backups - if you don't test them, they don't exist.

Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.

➕ show 1 reply
Betelbuddy • today at 9:08 PM

>> That is why all the cloud providers have multiple zones....and I'm amazed the big companies reported as down weren't using multiple zones.

Well then you should know that a Google Zone is not the same as an AWS AZ and many of Google regions are just firewalls across the same data center.... Despite all their SREs and Books and Phds...they dont know how to do infrastructure.

➕ show 1 reply
td521534f12 • today at 1:32 PM

GCP is kind of odd in that some hardware is only available in certain zones, and some hardware is technically available in multiple zones but the capacity is not evenly spread. I've had to implement infrastructure code for GCP that interrogated hardware availability by region and zone to place VMs, trading off multi-AZ availability where it made sense.

➕ show 3 replies
physhster • today at 1:53 PM

Google is PDT-centric, and never grew out of it. Everything is Mountain View time internally and it's so jarring.

➕ show 1 reply
InsidiousWombat • today at 2:45 PM

The fact they didn't immediately fail over doesn't mean they didn't had a plan.

The plan might've just been "if it gets bombed, we restore backups into other zone". Just waiting for it to fix itself for few hours mightv'e been acceptable risk.

➕ show 1 reply
crumby • today at 12:51 PM

Often there are irritating legal geographic limitations on where cloud hardware can exist for a given country. I can understand why, but there is this particular drawback of having all eggs baskets close together.

➕ show 1 reply
cavoirom • today at 12:20 PM

I know some banks out-sourced most of its infrastructure to external operators, and some operator I worked with didn't seem competent.

TacticalCoder • today at 7:38 PM

> ... I'm amazed the big companies reported as down weren't using multiple zones.

Yeah well, they drank the cloud kool-aid. The promise was that they'd go "cloud", and that everything would be taken care of.

Turns out: availability is worse than when those companies used to self-host and major cloud outages are happening all the time (at this point it's basically a meme).

And now the promise that "the cloud is simpler than doing it yourself" turns out to be not true?

I'm not surprised at all: these companies have been sold a lie.