alt Hacker NewsMany sites deliberately whitelist Google IP's for their search crawler and ignore crawlers who just uses the Http UserAgent of GoogleBot.
Many sites deliberately whitelist Google IP's for their search crawler and ignore crawlers who just uses the Http UserAgent of GoogleBot.