You’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it:

* For every Git commit, fetch the version of every file in the repository at that commit.

* See git blame for every file at every commit.

* Attempt to download the archive of each repo at every commit.

* Run every possible pull request search filter combination.

* Run every possible issue search filter combination.

* Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before.

Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.

Couldn't they just rate limit them? Are they literally using a new IP for every request?