I am not surprised...

I am unfamiliar with web caching proxies like squid [0] but I am wondering if that might be the most straightforward way to do this.

So use squid and then have a batch job that go through /var/spool/squid every day and update your web archive according to some defined filters.

- [0] https://www.squid-cache.org/