logoalt Hacker News

sixtyjyesterday at 9:11 PM2 repliesview on HN

The issue with scrapping is the intensity and volume of bots.

I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.

Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.


Replies

ccgregyesterday at 10:15 PM

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.

show 1 reply
2000UltraDeluxetoday at 10:23 AM

> I think that nobody would care if I use wget or curl for few pages

If only you were the only one doing it...