logoalt Hacker News

tailrecursiontoday at 8:52 PM0 repliesview on HN

Do the AI labs in the U.S. publish the AWS IP address sets that their crawlers use? Is that not an effective way to block those crawlers anymore?