And you think all this web AI crawlers will respect robots.txt. That era is gone.
The big USA ones do, and it would be madness for them to do otherwise.
But that's meaningless because 99% of AI crawlers are "bad bots" which ignore robots.txt and use domestic IPs to circumvent blocks.
alt Hacker News
The big USA ones do, and it would be madness for them to do otherwise.
But that's meaningless because 99% of AI crawlers are "bad bots" which ignore robots.txt and use domestic IPs to circumvent blocks.