How Uber Protects Against Retry Storms

It's not related to the article content, but I can't help but notice the "diversity" in authorship grades: a senior staff, a (regular) senior, and a principal. I suppose grades are a key mechanism by which corporates keep their hamster wheels running.

I'm thankful that, at my $WORK, we're all software engineers by title, from the lowly interns to the CTO. Still, everybody knows who's who and whom to talk to, depending on the gravity of the issue.

> Deepanshu Mehndiratta, Alok Srivastava, Vibhor Dhingra, Ankit Srivastava

It's not AI that is going to replace you, folks. Btw is it a coincide or has Uber become a family business now?

So, effectively if A → B → C → D and D is failing, C may retry D, but B and A are discouraged from retrying the whole chain.

This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)

Meanwhile Google keeps giving me “please wait, do not reload page” walls, so I ctrl-r as rapidly as possible. Or is that the human test and response?

I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point

I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?

Easy. Take a larger cut from the driver for each retry.

This feels like trying to reinvent Fibre Channel's flow control mechanism.

Token bucket is all you need

[dead]