I'm running some tests currently
https://deep.43z.one (how deep bots will crawl)
https://stalling.43z.one (how long they'll keep the connection open)
Ooh that’s fun! I’ll have to add some stats like this to mine too. Anthropic keeping connections alive for 2 hours is wild. I don’t think they fall into my trap as I believe they respect robots.txt, which is at least an improvement on Meta.
Ooh that’s fun! I’ll have to add some stats like this to mine too. Anthropic keeping connections alive for 2 hours is wild. I don’t think they fall into my trap as I believe they respect robots.txt, which is at least an improvement on Meta.