Queues Don't Fix Overload (2014)

Back in the 90s. I was part of a team working R&D project on flexible manufacturing. One of the central concepts was the use of buffers (also called decouplers) between manufacturing stations. These buffers were the space needed to place not finished products while they waited for the next station. Typically a conveyor belt. The main service is that both stations could run at slightly different rhythm, and they would not be causing problems on the other one. Either starving it or overwhelming it. Queues are pretty similar. You can have a consumer and a producer working and the queue in between prevents them from being blocked by each other. Thus I now see the main role of queues as decouplers.

Author clearly has a wealth of real experience, but I have trouble reconciling some of it to the “real world.”

Supposing that you have “too many” messages in your queue, commanding your frontend client to retry its transaction that would’ve added one more, instead of accepting and enqueuing one additional job, doesn’t seem to me to change much. Instead of creating a mess for whoever is in charge of those servers, the mess is created directly in view of the end user, who sees whatever you show them when their transaction is being retried.

Their point about the bottleneck being the real problem that must be addressed if loads are going to be sustained at such a high level is indisputable, though.

I think I would define the necessary rule as: the queue’s maximum size just needs to be greater than the spikes you expect, but that’s of course no insight, just a definition.

I have found queues to be incredibly valuable at solving situations where load has occasional spikes, but urgency of the jobs being done is low. For instance, every time a user views a piece of content you want to make sure that you increment a counter of how many times the content has been viewed, and you also want to touch the timestamp of when that user last did a thing. If that happens even two hours late, it’s probably gonna be fine. The thing that the queue pattern excels at in the realm of Web applications, especially, is allowing you to have an HTTP GET which can be served entirely by a Web worker that is only allowed to talk to a read replica, which allows extensive horizontal scale. Analytics and other incidentals can be handled async in background jobs (and indeed, in emergencies, load-shedding those ancillary things has barely any impact).

I recognize that all of this probably sounds “obvious” - but I have seen enough codebases that do synchronous writes during GET transactions that I would stop short of calling this “common knowledge.”

Once you've read this, pick up Harchol–Balter's book Performance Modeling and Design of Computer Systems: Queueing Theory in Action. It's a really good introduction to the depth of this topic, and you'll come out with superpowers you didn't have before.

Similar things apply to buffered channels in Go: buffering frequently hides deadlocks, etc, but you don't observe them until the buffer is full. So buffers generally should be kept either as zero or very small to be able to catch synchronisation issues early

Or you could just play Factorio.

Two past submissions with notable discussions:

https://news.ycombinator.com/item?id=39041477 - 18 Jan 2024, 153 comments

https://news.ycombinator.com/item?id=8632043 - 19 Nov 2014, 60 comments

Reminds me of this other queuing theory blogpost classic https://medium.com/swlh/fifo-considered-harmful-793b76f98374

TLDR LIFO (stack, not queue) is often a better choice for many workloads, despite violating our sense of fairness.

"People misuse queues all the time. The most egregious case being to fix issues with slow apps, and consequently, with overload. But to say why, I'll need to take bits of talks and texts I have around the place, and content that I have written in more details about in Erlang in Anger."

This feels like an unfinished gripe—summarize the key point now rather than promising a future deep dive. Backing claims with a concise example would make the argument useful instead of vague.

[flagged]