alt Hacker NewsI agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
Replies
jay_kyburz • yesterday at 9:34 PM
How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.
➕ show 2 replies
Well this is not what is happening in practice, Wikipedia / Wikidata, OpenStreetMap, OpenFoodFacts... All provide APIs and even a full dump of their database available to download for free, but no, the stupid bots still DDoS them 24h/24.