alt Hacker NewsWhy do this though?
It's like if someone was trying to "trap" search crawlers back in the early 2000s.
Seems counterproductive
Replies
integralid • today at 2:16 PM
search crawlers used to bring people TO your site llm boots are used to keep people OUT of your site, because knowledge is indexed and distributed by corporations.
➕ show 1 reply
Forgeties79 • today at 1:10 PM
Web crawlers didn’t routinely take down public resources or use the scraped info to generate facsimiles that people are still having ethical debates over. Its presence didn’t even register and it was indexing that helped them. It isn’t remotely the same thing.
https://www.libraryjournal.com/story/ai-bots-swarm-library-c...
➕ show 1 reply
Because of bots that don't respect ROBOTS.txt .
If you want an AI bot to crawl your website while you pay for that bandwidth then you wont use the tool.