> How to run shell commands in parallel while limiting their concurrency.
Depending upon the task, generate a Makefile and run it with:
make -j <concurrency>
> My original use case had to do with making a large number of web requestsThis is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.:
BASE := https://example.com/files
FILES := a.txt b.txt c.txt d.txt e.txt
.DELETE_ON_ERROR:
all: $(FILES)
$(FILES):
curl -sS --fail --max-time 30 -o $@ $(BASE)/$@
Or, to emulate one of the examples in the blog post: SHELL := bash
.SHELLFLAGS := -o pipefail -c
N := 1 2 3 4 5
REQS := $(N:%=req-%)
.PHONY: all $(REQS)
all: $(REQS)
$(REQS): req-%:
@curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \
-d '{"request": $*}' https://postman-echo.com/post | jq -c .json
Run with: $ make -O -j 3
> You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.Ah, yeah, that does put Make in the same bucket as `xargs -P` / GNU `parallel` and requires something more clever.