logoalt Hacker News

ed_mercertoday at 1:37 AM1 replyview on HN

> Honors robots.txt directives, including crawl-delay

Sounds pretty useless for any serious AI company


Replies

PeterStuertoday at 8:02 AM

What % of sites have a content update volume that exceeds what you can get respecting crawl delay?

If your delay is 1s and you publish less than 60 updates a minute on average I can still get 100%. Most crawls are not that latency sensitive, certainly not the ai ones.

HFT bots, now that is an entirely different ballgame.

show 1 reply