logoalt Hacker News

stefankayesterday at 5:52 PM2 repliesview on HN

Where does one find a good robots.txt? Are there any well maintained out there?


Replies

skrtskrtyesterday at 10:04 PM

Cloudflare actually has this as a free tier feature so even if you don't want to use it for your site you can just setup a throwaway domain on Cloudflare and periodically copy the robots.txt they generate from your scraper allow/block preferences, since they'll be keeping up to date with all the latest.