alt Hacker NewsCloudflare specifically has a block for LLM and AI training bots now.
Not sure of the effectiveness but it's there.
Replies
Minimal. I'm behind Cloudflare and 90% of the traffic is still scrapers. I don't think they're serious about the long tail.
I think the main thing Cloudflare is trying to do is block direct traffic from frontier labs and then start charging them for access. They might end up shooting themselves in the foot, as this simply empowers sketchy residential-proxy outfits to undercut Cloudflare and sell the data to labs for less.
It still only blocks "well-behaved" bots that have proper User-Agents and respect robots.txt, so it's largely pointless.
The problematic bots are all disguising themselves as Chrome and sending requests from millions of residential proxy IPs, and the only real solution to those is some sort of captcha or PoW page on first visit.
While they themselves announced an AI bot, the irony is palpable. In reality they just want to control who gets access to what, themselves excepted.