alt Hacker NewsYes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.
Replies
dylan604 • today at 2:23 AM
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
➕ show 1 reply
Common Crawl's dataset was downloaded in full 100 times in 2025.
We agree that it would be great if it was even more widely used.