This thread prompted me to look into this. It seems that all I need is a thin wrapper around whisper-ctranslate2. So I wrote one and am playing with it right now.

I'm finding language auto-detection to be a bit wonky (for example, it repeatedly identified Ladykracher audio as English instead of German). I ended up having to force a language instead. The only show in my library where this approach doesn't work is Parlement[1], but I can live with that.

On the whole this is looking quite promising. Thanks for the idea.

[1] https://en.wikipedia.org/wiki/Parlement_(TV_series)