logoalt Hacker News

andrew_zhongyesterday at 7:31 AM1 replyview on HN

I will add a PR to enforce robots.txt before the actual scraping.


Replies

messeyesterday at 11:44 PM

Or just follow web standards and define and publish your User-Agent header, so that people can block that as needed.

You're creating the wrong kind of value. I really hope your company fails, as its success implies a failure of the web in general.

I wish you the best success outside of your current endeavour.