some people don't want to be scrapped at all, and most bots ignore `robots.txt`