logoalt Hacker News

hoistbypetardtoday at 4:23 PM0 repliesview on HN

If you do that and don't follow robots.txt, you are blocked. If you do that and follow robots.txt, fine. That's all we wanted you to do anyway. Just follow the instructions that well-behaved scrapers are meant to follow.