alt Hacker NewsYou’re so right. I have a public-facing Forgejo server. Before configuring Anubis, scrapers were sending it about 600K requests per day. Copying and pasting from my blog post about it:
* For every Git commit, fetch the version of every file in the repository at that commit.
* See git blame for every file at every commit.
* Attempt to download the archive of each repo at every commit.
* Run every possible pull request search filter combination.
* Run every possible issue search filter combination.
* Fetch each of those URLs at random from some residential IP in Brazil that had not ever accessed my server before.
Afterward, it dropped to several hundred. Expect anti-attack features to keep getting stranger and more visible as scraper get still more aggressive.
Replies
That's crazy. So 600,000 someones are absolutely starved for data.
Or one someone with too much money and too little sense misplaced a decimal point in their ScraPy setup?
This matches my experience of running a public mediawiki server. The bots (mostly Facebook/Meta) will for every single change fetch every single page again, over and over.
I can’t tell if it’s incompetence or malice.
What do you use to monitor this? I don't really keep a close enough eye on my services to know what the traffic is doing (and haven't had any issues) but maybe I should start