alt Hacker NewsBecause the scraper is either impatient, careless or indifferent; and if they scrape for training data they don't plan to come back. If they don't plan to come back they don't care if you tighten up crawling protections after they have moved on. In fact they are probably happy that they got their data and their competition won't
> they don't plan to come back
To me the current behavior of those scrapers tells me that "they don't plan", period.
Looks like they hired a bunch of excavators and are digging 2 meters deep on whole fields, looking for nuggets of gold, and pilling the dirt on a huge mountain.
Once they realize the field was bereft of any gold but full of silver? Or that the gold was actually 2.5 meters deep?
They have to go through everything again.