logoalt Hacker News

h43z • yesterday at 10:11 PM • 1 reply • view on HN

I'm running some tests currently

https://deep.43z.one (how deep bots will crawl)

https://stalling.43z.one (how long they'll keep the connection open)


Replies

robotmay • yesterday at 10:51 PM

Ooh that’s fun! I’ll have to add some stats like this to mine too. Anthropic keeping connections alive for 2 hours is wild. I don’t think they fall into my trap as I believe they respect robots.txt, which is at least an improvement on Meta.