alt Hacker NewsThere are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
Replies
>We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?
How did you implement this? My site's getting hammered, any tips would be appreciated
> Most scrapers are relatively honest in some way shape or form.
Did you miss a "dis" in there?
One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1]
[1] https://www.gentoo.org/news/2026/01/05/new-year.html