alt Hacker News> Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.
DDOS?
Replies
If Outlook can go down because of DDOS then it's a much worse problem.
My guess: too many DC resources dedicated to AI, leaving insufficient free capacity to handle traffic spikes
Reminds me of this https://www.reuters.com/sustainability/boards-policy-regulat...
Nah, just a "Reply-All" chain at NVDA...
>reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load
This seems to be the smoking gnu, now what would it be ...
Probably self inflicted by some code change, as they mention "specific type of service traffic".
EDIT: Seems like now they confirmed that: "We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service."
As someone who worked in infrastructure for a large global CDN, this sort of overload issues happen all the time when trying to bring infrastructure back online after an issue.
First, you have every client retrying connections, whether automated or manual retries. This generates extra load over normal.
In addition, whichever systems come online first often get a ton of traffic sent to them, which can immediately bring them back down if you don't have ways to prevent it. You will get a wave of systems coming back online, a ton of traffic hitting them, and then falling over again.
Now, I am sure Microsoft is used to this and has procedures and systems in place to mitigate the issues (I know we did).
However, sometimes if the right combination of services are failing, even the strategies you had to avoid this sort of failure can themselves fail.