Lets take a moment to consider why that has started happening.

The lineage of every LLM of relevance today was the largest theft of intellectual property in human history. Every digitised creative work, whether in the public domain or not was taken, often outright pirated, with the explicit intention of being able to trivially produce compelling forgeries. Physical works are being enthusiastically destroyed to digitise them to further feed the hunger. And outside of a few court cases, the original creators across the world have been paid nothing for their work having been stolen and repurposed to put them out of a job.

Meanwhile, those same digitisation efforts have resulted in badly behaved scrapers from all comers absolutely thrashing the entire web, in many cases exponentially increasing the cost of hosting for smaller entities. Larger entities care less about that but are still enthusiastically walling their gardens so they can monetise the output of their communities rather than continuing to allow it to be stolen freely.

We are siloing our knowledge out of desperation. I'm sympathetic to the idea that knowledge should be free but that isn't possible when bad actors like Google and OpenAI are preying on the commons.

AI is already fully capable of solving the majority of/all captchas, you just aren't supposed to do it due to TOS.

I expect with time these kinds of things will change.

Not we, just Elsevier. As long as Academia sells its research, information will not be free.

Who says the agent was treated the same as you?

If the "AI" scrapers honored HTTP 402 I'd be happy to let 'em in.

Or necessarily depending on your viewpoint.