logoalt Hacker News

soundworlds • yesterday at 9:25 PM • 3 replies • view on HN

Update from Microsoft: "We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly.

"We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses."

https://www.abc.net.au/news/2026-11-13/microsoft-outage-affe...


Replies

EvanAnderson • yesterday at 10:20 PM

The 11:05:31 PM UTC update is is the first since the outage began that feels hopeful.

From the 11:06:31 PM UTC update:

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.

We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

---

The 10:11:39 PM UTC update is a little chilling. They don't know why what's happening is happening so they're doing experiments.

I do that with closed-source software when I don't want to bother trying reverse engineer it, but these guys have the code! Yikes.

I recognize that even w/ the code available there are likely behaviors that aren't likely to be anticipated because of the scale of traffic. Complex systems fail in complex and non-intuitive ways.

From the 10:11:39 PM UTC update:

Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

➕ show 3 replies
Rantenki • yesterday at 10:19 PM

Update as of 10 minutes ago implies that didn't work:

> Current status: We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

➕ show 1 reply