The responsible internet is dead. Every big actor on the internet is selfish now that there's money involved. And has been for 20 years.

Google only respected it because blocking Google from crawling your site used to hurt you more than it hurt Google.