alt Hacker News> Is it allowed to use automated tools to interact with any system?
I'd hope so, because that's what we're doing right now. Your browser is automatically speaking HTTP for you so that you don't have to.
Am I having a bit of a laugh? Maybe. But really, services should be user-agent agnostic. That's the whole "agent" part of User Agent and the founders of the Internet had incredible foresight to name it this way.
> Is it allowed to scrape data?
You mean, request data and receive what the other server voluntarily transmits?
> Are there any laws for this?
There was a court case that said the above is fine, thankfully, since that's how the internet works. There's probably other cases going on and I'm sure at least one of them will have some unfortunate tech-illiterate result that makes things worse for anyone who understands this stuff.
Replies
> request data and receive what the other server voluntarily transmits?
Taking your position to its logical conclusion implies that we shouldn't try to mitigate DDoSes either.
In many cases, what the other server voluntarily transmits has so far been based on the tacit assumption that a person, with person-level time and computational power, is doing the receiving. While in principle a machine could be doing it even in pre-LLM times, in practice many websites, including all the biggest ones, have implemented a wide range of approaches to try to curb machine access, starting with user agent checks and rate limits but by no means ending there.
The question is: Given the new landscape, where this assumption increasingly does not hold (because AI agents are increasingly able to simulate anything a person could do online), would those servers voluntarily transmit that data? In many cases, the answer is no.
> Your browser is automatically speaking HTTP for you so that you don't have to.
Yes, but it's not filling in the forms or clicking the buttons for me. HTTP is just infrastructure. Are LLMs infrastructure? Are we too maybe infrastructure? Where do we draw the line?
> You mean, request data and receive what the other server voluntarily transmits?
I mean to go over a large collection of publicly or privately (to you) available pages and parse and collect the data, with idea of using it in other purposes.
Regarding scraping, considering that this whole AI phase was built on illegal scraping, I don't think they can say anything now...
That's the whole "agent" part of User Agent and the founders of the Internet had incredible foresight to name it this way.
Now it's the Agentic User Agent.