Common Crawl's archive has metadata that says when each record (html file) was crawled.
But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.
alt Hacker News
But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.