alt Hacker NewsDon't we need more than an index of Archive.org because whomever controls the domain could robots.txt these out of existence if they wanted to?
Don't we need more than an index of Archive.org because whomever controls the domain could robots.txt these out of existence if they wanted to?
Archive.org mostly ignores robots.txt
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...