logoalt Hacker News

boredatomsyesterday at 10:12 PM1 replyview on HN

Can I suggest putting some text in the page that tells the bot what the more efficient download method is?


Replies

zbentleytoday at 2:28 AM

What makes you think that would accomplish anything?

These scrapers aren't LLM agents. They're distributed classical programs that harvest data which is later used to train an LLM. The LLM doesn't write the scraper or respond to individual scrape events. The entity training the LLM contracts someone, who contracts someone, who contracts someone to run a web scraper and send them the data.