alt Hacker News> If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user.
No; in this case you are not a user, you are a bot user.
Replies
you know what a browser is called by the web site? check the header that tells the version.
USER AGENT
not user, an agent on behalf of the user.
the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.
for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)
The best way to read the information on the internet today is via a LLM.
Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.
There is bot traffic initiated by a human and bot traffic not initiated by a human. I would want to serve the first but not the second if it impacted my cost/performance at all.
"Bot" access on behalf of users should be fine, even preferred. The missing piece is some micro transaction layer and some sort of attestation somewhere in the layer that the person driving the bot is not a bad actor.
Equating bot with bad actor in 2026 is Luddite behavior. Driving people to your site so you can serve them adspam or just make whatever operation they want to do 10x more difficult is the same.
MCP style APIs should eat the web. This doesn't mean the "open" web goes away. > 99% of the time I don't care for a domains particular FE at all. It's a complete waste not only of time, but resources and bloat. Pushing the contract into the agent should become good UX. Making things harder for good faith users should never be the goal.
I assume you are unfamiliar with the concept of user agents? Otherwise your browser would count as a bot user no?
And if not, what if it was a custom browser and not Chrome/Firefox/Edge?
Writing a script to fetch HTML is no different than writing a web browser. I think it's the scale of the operation that distinguishes bots vs human. The browser is the user's agent, but not the only one.
A web browser is an entity that acts on your behalf. That's why it's called a user agent.
They're just better at English now.
I think it's time for people to build a local database of every site they've ever visited, and then they can give their LLMs access to that. I might be willing to pass this data off to a company to store for me. Companies already store all my emails and money--why not trust them with this too?
Like, all the comments of this post would go into my personal database simply because I loaded the page, and it would help me find old information I've read, and could also inform LLMs I use.
This should be built into browsers.
What if I’m a web alerts company? I crawl your content but my clients are all end users who actually see your content at your site.
Scenario one: you use software to connect to their server and download a webpage. You are a user.
Scenario two: you use software to connect to their server and download a webpage. You are a "bot".
Make it make sense
Are you gonna police the means through which I'm browsing the internet? If you want to, you are free to put your stuff behind the paywall and give the key only to people who agree to obey your conditions. If you put it in the open you can't make conditions. That's what publishing means. Author can't make demands in what manner their book should be read.
I don't disagree, but there is a sliding scale here.
For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.
However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.