The same could be said of robots.txt

And anything else that might tell them not to access something.