alt Hacker News"While accelerating our migration to Azure," meaning, they will only solve problems if it helps them also use Azure more.
It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage, unless they have no reporting or their tooling is completely incompetent. If things can fall apart so easily, throwing more capacity at the problem won't fix it.
Replies
I’m firmly in the camp of “something stinks at GitHub” but
> It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage
In my experience, there are hard thresholds that get passed that expose hidden bottlenecks like this. A previous system I worked on we had absolutely loads of headroom by all of our measured metrics, but one day we filled a cache because the value hadn’t been tweaked in recent memory. Plenty of space on disk and in memory, but all of a sudden we went from a very high cache hit rate to a very low cache hit rate, and everything ground to a halt.
There's always a cliff, this part is fine. You sometimes know the cliff but often do not.
This absolutely can happen in large systems. If some part of the system is at capacity, then slightly increasing the load can cause it to fall behind and start accumulating a backlog.
These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.
>"While accelerating our migration to Azure," meaning, they will only solve problems if it helps them also use Azure more.
Look, I hate Microslop as much as anyone but you'd have to purposely misinterpret TFA in order to arrive at this interpretation. C'mon.
You’re torturing your own logic to make Azure the villain here. And it also sounds like you lack experience with capacity exhaustion. Things fail slowly, then suddenly.