alt Hacker NewsHis example of a web crawler stinks. I can get behind the philosophy, but his specific web crawler example at a minimum needs to retry, respect robots.txt, and rate limit per domain. So no, xargs + curl is a dumb example.
Replies
nimih • today at 2:56 AM
Based on bot traffic to my personal blog over the past year or so, the minimum viable web crawler apparently does not need any of those things (much to my own personal annoyance, to be very clear).
groundzeros2015 • today at 6:24 AM
Nah. It’s fine.
Replace curl with wget, get robots.txt support. Write timed-out URLs into a file for the next identical run.
The idea stays: you can jerry-rig a web crawler from pedestrian, readily available parts, and quickly test the idea for which you wanted to crawl the Web.
The point is that bash, xargs, wget, make, etc are well-understood, well-maintained tools. They are not even necessarily simple, but they likely contain very few unknown unknowns.