alt Hacker Newsjust their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
Replies
dylan604 • today at 2:14 AM
If you scrape 6TB from across the web or grab it from one place, it's still 6TB
It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.