Amateur hour? Not sure if you've lived through one of these cascading failures, but they are hard to design against, hard to load/chaos test for, and hard to recover from. The key is that this is a dynamic load, not a static load; and it's normally amplified by uncontrolled retry behavior by clients.

In retrospect, the cause and cure are obvious to everyone.

Newest update says that wasn't the issue after all. They say relatively small loads are actually causing servers to become unresponsive and have to be restarted.

They still don't know why this "core process" is maxing out the processor for no reason.

This sounds suspiciously what occurred when I was working at a large motor sports company. We had a major release of new year models coming out on the website. The deployment went super smooth and then in about 30 mins the site was crashing. Since we were a huge MS client, almost everything we ran was all MS top to bottom.

Site was taken offline, war rooms ensued. 48 hours later, an Ops team from Redmond was flown out. Another three days and they determined it was several legacy servers were going down and the load balancing servers were getting hit so hard, they also failed. It was a cascading effect that killed everything.

CIO was pissed. The company got absolutely lit up on every social media channel for their ineptitude. The CIO vowed it would never happen again, they were upgrading the entire network and infrastructure to avoid it happening again. Fast forward six months to the next major release. Same thing, same panic, same cascading failures because servers were collapsing under the traffic to the website. Another war room, another Ops team flies out. Two days later they figured out it was some buggy code that was causing it.

This after 8 rounds of load balancing testing to the tune of 10X the amount of traffic the site sees on these major release dates without so much as a hiccup.

I left two months later, but the overall feeling was the same - total amateur hour from Microsoft after our company was gifted major discounts to go all in with their hardware & network gear. The company found out the hard way it wasn't a smart decision.

The scary part was this some 20 years ago - seems like not much has changed considering how frequently Github is down now too.

The people writing these updates are very obviously not technical and are least one step removed from the actual work being done to fix this.

I can't stand it when I see armchair analysts making claims like this.

I'm sure it's more complicated than that.... but from my POV, which is extremely limited and informed by what others online tell me.... that this is amateur hour

What would it take for it to be professional hour?

At Microsoft every hour is amateur hour.

Copilot hour!