It's documented here: https://developers.openai.com/api/docs/bots
- which one is to stop you from hammering small servers with hundreds simultaneous connections?
- why don't you just respect existing robots.txt that apply to you already?
- does every LLM scraper seriously think the onus to opt out from the EVERY SINGLE SCRAPER is on the webmasters/owners?
alt Hacker News
- which one is to stop you from hammering small servers with hundreds simultaneous connections?
- why don't you just respect existing robots.txt that apply to you already?
- does every LLM scraper seriously think the onus to opt out from the EVERY SINGLE SCRAPER is on the webmasters/owners?