It affects science too (and there you'd want solid archiving as much as possible). Increasingly, meta-data is full of errors and general purpose search engines for science are breaking down, including even things like Google Scholar. I suppose some big science publishers are blocking AI bots too.
> I suppose some big science publishers are blocking AI bots too.
That's a travesty, considering that a huge chunk of science is public-funded; the public is being denied the benefits of what they're paying for, essentially.
Thank god for pubmed and deterministic search operators.
Google ruined its own search engine on top of that as well though.
We are increasingly becoming blind. To me it looks as if this is done on purpose actually.