alt Hacker NewsPut a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.
Replies
jopsen • yesterday at 4:56 PM
They allow you to diff commits, which is an awesome feature.
But if bots a crawling diffs between all possible commits it's crazy.
CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.
➕ show 1 reply
grep_it • yesterday at 4:46 PM
Did you read the article?
“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
➕ show 1 reply
This is not meaningfully static content. Look at the charts in TFA. There is a combinatorial explosion of distinct URLs which the crawlers can and do request.