alt Hacker News750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour
Millions per hour is tens per second though; perhaps the fix is performance improvements
Replies
andai • today at 3:10 PM
Yeah we can just rewrite the web servers in Rust ;)
That'll be great until.. they rewrite the scrapers in Rust! Then we're really hosed!
Jabbles • today at 9:01 AM
1,000,000 / 3,600 = ~278
We have put in a number of performance improvements, yes. The nice thing about those is that they also make the site snappier when it's not under load. Right now we have just over a million items in our CMS, plus our publicly available mailing list archives, which are much larger, even if they're less frequently referenced.