logoalt Hacker News

Deathmaxyesterday at 1:40 PM1 replyview on HN

Except that officially, that is not what they do? It's a dick move not to put the two usecases under separate user agents, but their documentation says you're free to block Google-Extended via robots.txt which is used for training and grounding, while still being included in the search index.

> Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search. https://developers.google.com/crawling/docs/crawlers-fetcher...

Exclusion from grounding does mean that your site won't get sourced in the AI overview, but I'm not sure what the click through rates are like on those.


Replies

mysterydipyesterday at 1:48 PM

AI crawlers, famous for respecting robots.txt ;)

show 1 reply