alt Hacker NewsTangential but all the pages I see protected by Anubis are just static HTML. But bot traffic is only harmful on dynamic pages right?
Replies
Others have answered your question. A lot of the sites listed here https://anubis.techaro.lol/docs/user/known-instances/ have dynamic content; Git web interfaces in particular (Codeberg, the Linux kernel, FFMPEG, and more are on the list) are vulnerable to poorly or maliciously configured scrapers.
Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use. It's just stuck brute-forcing files or something, sucking up loads of bandwidth downloading the same image for no reason.
It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.
The anubis docs website is used as an example of how anubis works.
Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.
some people don't want to be scrapped at all, and most bots ignore `robots.txt`
no, absolutely not, any malicious or dishonest bot traffic is harmful, generating unnecessary transfer and literally stealing data
I'm not a fan of Anubis for various reasons but the idea that bot traffic in only harmful with dynamic pages must die. CPU (yes, even to serve static pages) is not free, bandwidth is definitely not free. There's an idea that serving a static page to a bot has a marginal cost tending to zero, but it's never really zero and serving them by millions definitely has a cost.
Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.