logoalt Hacker News

cmeacham98 • today at 1:20 AM • 2 replies • view on HN

Correct. Example snippet from the nytimes.com robots.txt:

    User-agent: archive.org_bot
    Disallow: /

Replies

mjmas • today at 10:41 AM

Is there a difference between that and User-agent: ia_archiver ?

joecool1029 • today at 2:30 AM

Which they don’t respect. I’ve had it for my blog for years and they still added it to wayback machine, see my last comment for their official announcement of the ignore robots.txt policy, it is not new.

➕ show 1 reply