alt Hacker NewsYou need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective.
From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirely sure but I believe google has two categories of scrapers, the regular one and a new one for AI
Replies
So it could just be 1 guy with scrapy and infinite vc money
Still not sure about the "who" question, but here's the scale of money changing hands:
> Residential proxies are everywhere, so why did proxy DDoS attacks mostly come from the U.S.? The answer is money. If you are committing fraud or circumventing content restrictions, a Russian IP address gets geo-blocked instantly. A fresh U.S. residential IP address (especially one behind carrier-grade NAT and harder to block individually) is “gold.” Customers pay up to $95 to lease a single U.S. residential IP address for 2 weeks (versus $0.30 for an Eastern European IP). Compare that to your own ARPU per subscriber and sit with it for a second. When an individual IP is worth more than the customer relationship behind it, you don’t have a technical problem. You have a market problem.[1]
1: https://www.nokia.com/blog/one-year-later-the-residential-pr...
if they have agents write the scrapers you better believe they're playing every evil trick in the book