Hey, I'm Nicolas from All About Berlin. This post is a few months old. I went through a psychological rollercoaster since I wrote it. Things got worse, but so far, I can still live from the website. Next year might be my last year running it full time.
I'm traveling at the moment, but I'll grab a drink and read this thread later today. In the meantime, feel free to email me. Reader mail would be a lovely break from the flood of LLM-generated outreach emails.
People are being conditioned to think that all the answers come from AI, or that AI knows everything.
But AI is only getting (inaccurate shadows of) the info from these sources which it is simultaneously working to kill. (Websites like this, StackOverflow, human written code)
What happens when there are no more sources and everyone just expects AI to answer everything or write everything? Recursive self-improvement? I doubt it.
The A.I. genie is probably never getting back in the bottle. How I would like for this technology to just not exist.
The hackernews frontpage would be sooo much nicer again as well. I used to browse hackernews being amazed and (pleasantly) surprised about the cleverness and imagination of people. but these days you have to look for the nice stuff with a lookingglass between the onslaught of breathless AI stories.
I just can't be bothered anymore to be amazed about what AI can dow now. I mean, it is amazing of course that we now more or less live in the asimov sf world. But it the kind of amazement I would very much prefer to have stayed a fantasy.
Please may I have some less.
This is the massive elephant in the room; AI needs human-made work, published openly to further develop, and simultaneously heavily discourages human-made work to be published openly, either because the people who previously did it now use AI to do it or because the people who still do no longer have any incentive to publish it openly (and lets be honest, the incentives were pretty meagre before).
But craziest of all is that the best plan this trillion-dollar industry can come up with is "we'll figure something out later". Maybe we will, maybe AI will be able to train off synthetic data or we'll achieve the perfect model that never needs improving or we'll simply hit a wall with how much training data we need, but for a problem that an existential threat to LLM's, that's a lot riding off a few maybe's
The paradigm shift we are living through is about leverage. It's akin to the printing press insofar as guilds literally lost much of the leverage to control pricing when printers would give away "how to do things" in pamphlets. Reselling the information the guilds had developed.
This will be economically devastating. It was back then. The question changes to how value can be created moving forward. The world in which information becomes free -- in literally every case -- creates a more prosperous world. How and whether that prosperity gets distributed equitably is the concern.
The fact that the tech industry has not unionized, because of their profound leverage over the last 30 years is unfortunate, but times have changed and we need to move forward. Businesses built on scalability may die, heck the open models my even kill off the brand name models.
People will still need to buy things. Developing ways to provide those things will still be valuable. Creative destruction marches onward.
It's unfortunate for 'good' website creators/maintainers but many people would prefer an AI summary of multiple peoples views on "getting a bank account in Germany" generated from lots sites on the internet than clicking through links of content of unknown quality and possibly outdated information that may just prioritize the banks with the best affiliate commission.
The vast majority of HN has always been pro-ad blocking, arguing that the user's wishes (to see no ads) override the content creator's interests (getting compensated for their work). The HN consensus is also generally, but less strongly, in favor of the primacy of the user-agent over the site in all other ways.
But AI summaries are pretty much indistinguishable from that. The search engine is the user-agent in this case. The user's actual desire is to get the information, and their agent (the search engine) is providing them an easier way to do it. The only differences are that it's on-by-default for 90% of the users rather than opt-in at 25% market share, and that there's no pro-forma page fetch of the content that could in principle be rejected by the site.
And let's be clear: users clearly do want AI overviews / AI mode. We might not understand why the users want it, but Google would not be implementing it if they didn't have to. There's a lot of reasons for why the status quo would have been preferrable, while the only real reason for a change is that the status quo was not tenable and they would have lost share to the sites giving users what they wanted.
The situation is a lose-lose, but it was also the logical outcome of saying that the end-user's wishes have total primacy over the other stakeholders.
I know of someone who lost their job in journalism, not because they are using AI to write but because the site is completely starved of traffic now.
Long term, with the current state of affairs, why would people continue to create online if it just feeding the machine? Surely we are not going back to paper?
That said, Private Eye never went online and is a successful publication.
To be fair, the website in question is highly recommending banks with an affiliate link first - banks that are in hot waters for violating anti money laundering regulations. If your business model is so dependant on a third party you have absolutely no control over, it wasn’t really a good business to start off with. And yes, LLMs will eat the lunch of any information-only sites that don’t anchor their audiences somehow.
May I propose an alternate way of thinking of this?
Internet advertising is a bubble; ad blockers were the canary in the coal mine; and AI summaries are the straw that broke the camel's back.
In this case, the "mom and pop" websites like All About Berlin probably need to find a different revenue source or publishing model. This, BTW, is "normal." In years past, "All About Berlin" might be an annual printed guide, a monthly magazine, a quarterly pamphlet, ect, ect.
All about berlin is an incredible resource. Helped me a lot when I moved from the bay area to Berlin and also with all the constant visa procedures post move.
I can recommend signing up to his newsletter for berlin events - https://allaboutberlin.com/newsletter.
I'm working on something[1] similar to All About Berlin but for Hamburg. Instead of going with a website, I went with an app. Sure, it is hard to reach more people but at least, I don't have to deal with this kind of situation and I can create a much better experience.
People (authors etc) are understandably upset about this. The models CLEARLY trained on data where they didn't own the IP. It just sort of happened too fast for laws to catch up, and now it just is what it is.
But let's assume for a moment that laws caught up. That courts decided that "no, you can't train an AI model by just scraping any content you want, and if you serve an AI model that suggests being trained on such content, you're also breaking the law".
What would happen? This seems like it would be a cataclysmic change to the economy? We'd suddenly "outlaw" several of the world's largest businesses, and the quality of AI models would jump backwards several years (Which on an AI timescale is like a century)?
But what if this is the "right" thing to do? Are we incapable of doing what's "right" now? Did the AI companies' bets on being quicker than legislation pay off?
The thing about the Google AI overview that I find most baffling is how self-defeating it is. You kill the traffic to sites, the sites die, now you don't have anything for your AI summary to pull from. Also Google is an ad company, they're literally removing their own revenue sources by reducing clicks.
Relying on Google for traffic was never a great strategy and I learned this the hard way myself. I have a website that produced ad revenue for very little work and used just make a big chunk of my income until about 15 years ago when Google traffic just collapsed, and that was that.
Its not AI, its big tech control that is the central problem.
I respect your work and I can completely understand your feelings. With that, I have but one advice.
Sue.
If they reuse your content, lawyer up and make them pay for it. Nothing will happen unless people stand up for themselves.
I recommend everyone try the example search for themselves (best german bank account for foreigners) and look at everything in the first SERP.
On DDG I get All About Berlin as #6. All the sites above it appear to be LLM generated. Everything is affiliate slop.
On Google I get All About Berlin #2, below Reddit. It only gives me 7 standard search results and a lot of other garbage. Everything besides Reddit is affiliate slop.
On Kagi I get All About Berlin #1. Out of the other 25 results one is Reddit, one is Facebook, and the rest seem to be all the same list style as All About Berlin with a mix of clearly LLM generated content and some that may have been but isn't clear. It's all SEOed.
This is a case where all the results are bad. I don't ever want to be recommended something from an affiliate site. It's just another form of advertising. I also don't want to see any LLM generated sites. None of this is organic content. It's all SEO sites to sell you stuff.
Intellectual property rights came into being to address the free-rider problem — patents, trade secrets, copyrights.
We might need a new form of IP right.
But to restate the obvious: A new form of IP right would likely require legislation — at the federal level. Therein lies the problem: Pretty much all elected officials spend (too) much of their time dialing for dollars for their reelection campaigns. That can make them reluctant to enact, or even endorse, measures that are opposed by their big campaign donors.
I have to say from the bottom of my heart: Thank you for making my 4 year stay in Berlin more bearable.
Some kind of reparations?
Maybe AIs could pay attributed sources, similarly to the way musicians license their samples.
To be fair, Google’s been suffocating search for years, before AI. Actual search results have been pushed below the fold for a long time, if you happen to not use an adblocker.
Think about what's coming next after the mega corps have assimilated all data in their private vaults. The next step is to go on q crusade to get the free Internet and the data they feasted on nuked from the web so that people have no choice but to rely on them.
You can expect something like browsers (chrome) refusing to load "unsafe" content i.e. material from the free Internet.
The usual advice is "do GEO now" and optimize to get cited in the AI answer. I'm not sure that saves a site like this. Even when you get cited, almost nobody clicks the link, because the whole point of the answer is that you don't have to. Being the source the AI uses and being a business are separate things now.
Switched to DuckDuckGo as my main search engine some years ago. Not even for ideological reasons, the results were just better.
Regardless of who provides what resource, the fundamental question is: is a static guide a better way to support people who want to learn about Berlin, or is the AI bot derived from that static guide better. Once you remove hallucinations from the evaluation (2024, what a crazy time, right?) it's pretty clear the bot wins, right?
That sucks for the content creator, I know. We're heading for a crazy pace of change, similar to but more intense than when Walmart was busy killing downtown shops, or when Amazon was busy killing bookstores and other internet retailers.
At this point, Google must be aware that:
- They had to introduce AI Search Overviews (competitors would've killed search altogether; now they have some sort of control).
- AI Search Overviews won't work without content creators.
- Content creators won't continue if AI Search Overviews keep eating their lunch.
- They have to come up with an incentive model for content creators to keep doing the things that power AI Search Overviews.
Time will tell.
Newspapers aggregated public info and got wiped out. What would make you think online info aggregators wouldn't get wiped out too?
If you aggregate something, AI will aggregate a thousand times better than you
It's time for horseback messengers to understand that cars and planes deliver info faster and better than them, and that applies to ALL branches of work
If it can be replaced by AI it will be, so be prepared and don't complain
Google gets a lot of stick on this topic as in this article. But it’s all of them. People “search” just as much in Claude and ChatGPT.
Can't people use cloudflare to block AI bots? It has a setting and google says it respects it if one does not want AI training.
Wow at that screenshot. And that's what everyone is using now? Surely it can't be that bad on every search.
Recipe sites too. The ai summary is usually a 100% copy of the top recipe result in the links, but stripped of extraneous info.
> How do you combat it?
Aside from the bot countermeasure arms race? Vicious litigation and collective bargaining/lobbying. No way around it, there never was. Unless to you "rolling over" is an acceptable, or even necessary, solution for your particular case, or in general.
This is one of the reasons why I try to ignore the AI overview and go straight to the links below.
I unashamedly place 100% of the blame of this on people who continue to use AI. Stop. It is ruining everything.
Everything you do is a choice. They say they'll fire you? Will they? Have they fired anyone for refusing to use it?
They can't fire everyone. Stand up and be counted.
We track the W3C's TDM Reservation protocol uptake, and I'm surprised it's so low, we only see it on 1,113 sites ( last month ), 1,876 sites this month.
https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-... https://theweb.report/#ai-web
Dead Internet Theory on steroids in here.
World is changing. Websites once killed the need of magazines and many types of books too..
I rather look at it as an opportunity to demonopolize Google as I do exactly the same (overviews and insights) I just find Google Gemini obsolete as I find better AI solutions where I can check for knowledge and information.
Open Web has a lot of content that was crawled before and I really hope that all that SEO/Traffic hunting nonsense stops to benefit Google towards major regression. They are so 00s company and we're heading towards 30s of XXI century.
Web is dead anyway.
All About Berlin is amazing, it has helped me a lot when moving here, there’s immense value in having a curated, regularly updated, always reliable source of information. I hope it can survive and also hope similar projects can survive.
Said it three days ago
>Google & friends no longer passing traffic through to the sites (AI summary style) feels like it may genuinely kill the internet.
Happening to me as well at BookDNA.com :(
Super frustrating to see Google crawl all our hard work, and then stop any traffic to us.
I was expecting to read about AI scrapers but it's not, it's about Google taking away their traffic by providing AI overviews. Something that's happening to many websites.
Have you tried providing poison data to Google?
its important to distinguish between training and agentic search both have similar effects but with very different solutions. for training we need licensing rules that apply to small content not just the studios that can afford the lawyers, websites provide payment and licensing terms for training access and training systems decide if they want to skip the content or pay. for agentic access its hard to imagine an alternative to micropayments similar to what cloudflare is experimenting with. i dont think a future where search results or recommendations will be visited directly much again is realistic.
It’s a snake eating itself type of thing, it reduces ad revenue and the investors already flinched on the last earnings report.
Never blame the technology. Focus on the people, or in this case the corporations wielding that power. It is in their best interest not to cause an extinction event of the ecosystem they thrive on. Currently they (Google etc) are merely leaching on the ecosystem without giving anything in return (destructive). Ideally, you reach symbiosis where there is a net positive result for both sides, ie. enough money flows back one way or another to the content creators you depend on. Either way, eventually this will come back to bite them, but in the mean time the ecosystem, maintained and built by real people, are the ones feeling all the hurt and it's a very sad state of affairs.
We need a new internet that can be protected by LLMs reading or writing it. And in general find a way to isolate the world from big tech by creating competitive products. I feel like if a lot of talented people got together we can do this.
More broadly, AI is killing the affiliate link and the ad-supported business model. As an AI user I love using AI, because it gives me what I seek (answers) and it takes a minuscule amount compared to what the previous systems took.
While I have never used it, All About Berlin seems like an exceptionally good resource, not showing ads, but rather making its money by driving traffic with good information and having partnerships with affiliate links.
Other than this, most of the ad-supported web I greatly dislike. Usually the noise drowns the signal, because websites competed not on the goodness of information, but the SEO rankings. I have been blocking ads since 2005 and I never used affiliate links either, so from my very narrow individual perspective, I was never a valuable visitor in the first place.
It would be great if the quality sites like AAB could get another stream of revenue that makes them sustainable, even in the AI age. I'm afraid they run into a similar problem to monetizing open source software. At the end of the day, if the information is out there, people and corporations will take it, and not contribute anything back. That won't change, so it needs to be factored in somehow.
As always, Google never owed this site any free traffic. The author was lucky to catch that free money while it lasted.
Wecome to the club. I managed a somewhat successful community website from 2003 and by 2018 it became a shadow of itself as users flocked to Facebook, Instagram and the likes.
It is nothing new in this case. It hit others with updates to Google's search results before.
The problem is, he is dependent on Google's traffic. This is a big weakness in his business model.
How do you all feel about the situation?
Is it just okay that these humongous companies just took all our work without paying us a dime?
Is it okay that they are replacing our jobs with tech that would never have existed had they not taken all that work? (or would have moved slower)
Do you feel good supporting these companies by giving them money every month?
What do you think will happen if these companies become profitable? Do you think it will have a positive effect on our society?
Personally I feel that I'm living with severe chronic cognitive dissonance - and have been for many months at this point. But I'm not sure what I can do, I have rent to pay and children to feed.
Do you all just not have this feeling? How do you combat it?