logoalt Hacker News

InsideOutSantayesterday at 8:08 PM0 repliesview on HN

> You can opt out from Google scraping you though?

You can't. Google will ignore robots.txt in some cases (e.g. "The REP isn't applicable to Google's crawlers that are controlled by users (for example, feed subscriptions), or crawlers that are used to increase user safety (for example, malware analysis)").

robots.txt is just a suggestion that Google loosely follows.

https://developers.google.com/crawling/docs/robots-txt/robot...