alt Hacker NewsI just complained to them the other day! They were scraping our weather website to no end, very much including the disallowed path prefixes.
Did end up just adding them to our WAF blocklist, which is weirdly ironic - hosting on their infra & using their services to block their AI scraper...
Replies
adrianvi • yesterday at 11:22 PM
step 1: create the problem, step 2: sell the solution, step 3: profit
I hope you leave it on the WAF. If they're only just deciding to respect robots.txt, which has been internet infrastructure forever, then it's probably still incredibly amateur software with 'Amazon-priorities' rather than 'responsible internet traffic' priorities.