Even companies like OpenAI segregate their training crawlers (GPTBot for general corpus training; OAI-SearchBot for their search index inclusion; and ChatGPT-User for user-driven agentic browser use, etc).

When even OpenAI is more respectful of intellectual property and website owner control than you, that's a problme.