logoalt Hacker News

djoldman • today at 8:27 PM • 1 reply • view on HN

Just FYI, the bigger companies all allow you to block crawlers via robots.txt:

    # Block Anthropic (Claude)
    User-agent: ClaudeBot
    Disallow: /
    User-agent: Claude-SearchBot
    Disallow: /
    User-agent: Claude-User
    Disallow: /

    # Block OpenAI (ChatGPT)
    User-agent: GPTBot
    Disallow: /
    User-agent: OAI-SearchBot
    Disallow: /

    # Block Perplexity
    User-agent: PerplexityBot
    Disallow: /

    # Block Google's AI Training
    User-agent: Google-Extended
    Disallow: /
    User-agent: Google-Extended-Factual
    Disallow: /

    # Block Microsoft's Search & AI Crawler
    User-agent: Bingbot
    Disallow: /

Replies

frereubu • today at 8:50 PM

This is good to know, but a bit of all-or-nothing. It's a shame that, for example, Google doesn't support the crawl-delay field so you can tailor their crawling to your setup: https://developers.google.com/crawling/docs/robots-txt/robot...

I presume it would also cut you off even more from referral traffic.