logoalt Hacker News

zendist • yesterday at 7:14 AM • 1 reply • view on HN

Regardless. You should still respect robots.txt..

Replies

andrew_zhong • yesterday at 7:27 AM

We do respect robots.txt production - also scraping browser providers like BrightData enforces that.

I will add a PR to enforce robots.txt before the actual scraping.

➕ show 1 reply