alt Hacker NewsGood. Google's approach here is manifestly predator, unfair, and IMO illegal. They deserve to be in court for this behaviour, and mandating owners give consent for AI training or drop out of Google; which is just a non-starter because they're a search monopoly.
That's exactly what antitrust laws are supposed to do, and I hope at least EU regulators take action. Every single Googlebot crawl in your access logs is a trace for damages.
Replies
I think it's bad, because everybody is desperate to hold onto every last bit of google search traffic they can, so they're going to allow training to do so. Google's predatory, unfair and illegal actions will continue as they have with a few $100 million slaps in the wrist from the EU and a few more white house dinners for their CEO.
Except that officially, that is not what they do? It's a dick move not to put the two usecases under separate user agents, but their documentation says you're free to block Google-Extended via robots.txt which is used for training and grounding, while still being included in the search index.
> Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. https://developers.google.com/crawling/docs/crawlers-fetcher...
Exclusion from grounding does mean that your site won't get sourced in the AI overview, but I'm not sure what the click through rates are like on those.
EU will write some strongly worded letter.
Saying this as a European who is pro EU.
Why should they?
I don't think it's the Google bot DDOSing people's infrastructure for AI training...
This had me in disbelief since the minute I saw it: Google's "AI overview" presumably trained on content from other websites, disincentivizes users from clicking through to those websites..
How is that not conflict of interest??
> mandating owners give consent for AI training or drop out of Google
Huh?
Search and AI are hand-in-hand.
They both rely on embeddings. (Unless you still do keyword-only search, but that's not as good.)
How come you say "Good." and you are near the top of the comments but I say "Good." and get flagdead?
The irony here is that the people blocking all other crawlers are the ones shoring up their monopoly. If you can't block Googlebot because you need the search traffic but you block everybody else so that nobody other than Google can index your site, how do you expect to ever get any search traffic that isn't from Google?