alt Hacker NewsA couple of years back I worked with a company which maintained specific data which was the main traffic driver on that page. Google approached them and wanted to pay for the rights to get the data and display it on top of the search results, a feature which was fairly new back then.
This was an interesting dilemma because it was very clear that the money was way less than the loss in ad revenue due to traffic drop, but it was also clear that if we wouldn’t take the deal, a more desperate competitor would, which would result in the same traffic loss but without the extra google money. So the company took the deal.
History repeats itself here, with the difference that instead of paying for the data, the ai crawlers simply take it for free.
Replies
Then we go and pay to Cloudeflare or other cloud providers you host sites with to prevent Google from scraping.
Let the big guys fight it out.
There is money in protecting the websites. If you host with OVH they are interested in you making money so you can pay them.
This reminds me of Walmarts squeezing strategy with all the manufacturers. Business with us at the price we say or out of business.
A similar dilemma presents itself when blocking AI spiders.
You’re free to block them, but the websites cloning your content won’t. So either way they’ll get the content they’re after.
Worse, when/if the time comes that LLMs source their claims they’ll refer you to the websites that cloned your content.
That doesn't feel like a repetition at all? You said that the first time there was "traffic loss but without the extra google money", but that this time there's no extra google money either way.
"Nice data you got there, it'd be a shame if something were to happen to it"
That's the problem with the current monopolistic system, the money won't go down the stream, it's like a dam. One big dam owned by a few people is worse than many small dams