logoalt Hacker News

saghmtoday at 2:30 PM2 repliesview on HN

Couldn't they just rate limit them? Are they literally using a new IP for every request?


Replies

anematodetoday at 2:43 PM

There are indeed scrapers which use tens of thousands of distinct IPs, and so rate limiting them isn't a solution.

show 1 reply
kstrausertoday at 5:06 PM

In my experience, yes. At the peak of a scraper flood I was dealing with, I'd say about 90% of the traffic was from a unique IP. I'd never seen anything quite like that before.