logoalt Hacker News

numpad0yesterday at 12:45 PM1 replyview on HN

Wayback Machine full archive is like less than 50PB. Let's say you could strip multimedia and remove every patterned data to compress that into about a petabyte. The per-bit cheapest disk right now is consumer Seagate 24TB(SI; 21.8TiB usable) at ~$500, or around $1200k for just the disks.

Doable if you had couple million dollars to burn. Cheaper than private jets new.


Replies

nijaveyesterday at 8:48 PM

Seems like the most practical path is some kind of distributed peer to peer contraption where you can allocate some storage and optionally participate in crawling.

I imagine if set some constraints you could get index size down quite a bit but I still suspect it'd be hard keeping up with content churn.

Edit: Looks like enwiki bz2 is coming in around 46Gi which isn't too bad considering the amount of content it contains.