alt Hacker NewsAnd by linking to your site from other websites, by not requiring authenticated human-only login, and by not blocking traffic from all referrers, you agree to allow my bot to access you site at zero cost.
And by linking to your site from other websites, by not requiring authenticated human-only login, and by not blocking traffic from all referrers, you agree to allow my bot to access you site at zero cost.
I have clearly communicated the constraints of use on my site using robots.txt, which is the established standard for communicating things like these. You are expected to fetch it and figure it out. It is not necessary to build walls and lock doors when a no trespassing sign is clearly visible.
Seriously, though, making robots.txt have the weight of law would be a massive improvement for the ecosystem in general.