I have tried a ton of stuff (Cloudflare, Anubis, fail2ban...) to combat the bots lately, but they all have their downsides, and users complained too much, so I just gave up. You simply can't win this arms race.

So I have resorted to performance improvements on my server and serving every single request. Things are way smoother now.