> What scares me about this new AI mode thingy

What scares me is the rampant inaccuracy. In my experience, the AI responses are wrong about 65% of the time. I just did a search today about an error talking about a disconnected link between apps, and Google AI result summary told me that the error was related to my pulling a USB drive too quickly in windows. The ONLY word similar to my query and that AI response was the word "disconnect". Everything else was clearly about the SaaS apps.

I have people coming to me, asking me questions, then telling my Google told them something else, so now I have to waste time convincing them that it's wrong. Over the past 2 years AI has done nothing for me but complicate my work life.

And of course, this could be because the model is crap, but it could be because they want me to keep refining my query over and over for more ad views. Either way, it's a terrible experience.

Yup, I was looking up a pair of IEMS vs another pair of IEMs. It said option A is overall better, when really it was just reciting a single person's opinion. I've been aware it will summarize only a single source and present it as an aggregation of many opinions, but it stood out to me how matter-of-fact it was that the one was definitely better than the other. I simply wanted to find forum discussions on people's thought and wasn't influenced by this AI blurb, but I think seeing an answer at the very top state so matter-of-factly that one is definitely better and present it as though everyone thinks that will definitely influence a lot of people. It makes me wonder how "gameable" this will become...

Indeed - just earlier this week I read Google AI summarize a query about testosterone, citing 3 sources. The first citation was a link to a NIH study (or of similar repute). Ok great. The second? Two spam (and explicit) websites existing solely to sell penis enlargement pills.

What was worrying is only some of the claims were supported by the linked study, and most of the response content was drawn from the spam sites.

This problem is not limited to Google, it's the core value of mass-marketed LLMs, or isn't it?

Without "random comments", Google wouldn't have anything to say about "does an air purifier help my asthma, if yes: which one?" or "find the problem with this Hibernate annotation".

They also don't make much effort to exclude sloppy sites, to the contrary, they made way more efforts against SEO spam in the time when Google was a search engine, not trying to be an AI "oracle".

I think their end game is that the only metrics relevant for ranking sources are:

- agreeability (works well as a proxy for correctness with many questions!)

- originality, but not in a scientific sense, just to prevent model collapse

- legal factors such as preventing false health claims or similar things, as long as there is legislation against this kind of thing

And half the time, the sources turn out to be sarcastic jokes on reddit.

I’ve noticed this too. A single result can determine the answer it gives. And removing the content from its context makes it harder to assess. Suddenly it’s “Gemini said …” rather than “some guy in the YouTube comments said”.

I ran into one that kept referencing "people", but then I found that it was a single Reddit thread from a couple of years ago about a relatively small and obscure foreign city with 2 replies.

The scary bit is the use of the term AI. The "I" implies critical thinking.

For models trained on a corpus of groomed data, the "critical thinking" bit is baked into the work of grooming the data and how it is trained. And someone is thinking critically about both so as to make a good model.

Now, every damn thing is called AI no matter where it is getting results from.

Are modern models super handy? Absolutely.

But calling it AI implies a lot more critical thought than is actually happening!

Edit: took the time to write a shorter comment.

I love asking AI about blatantly wrong opinions but by people it thinks must be an authority.

To not make this political, let me give you a game example. Right now the dota 2 fandom wiki is abandoned, and it has been vandalized with covert shitposts. One of them was the addition of a 4th attribute called Charisma, which is completely fake. If you ask AI's "What are the main attributes in dota, according to the official wiki", the dumber AI will fall for it, but the smarter AI will know it's wrong, but try hard to hallucinate some sort of valid explanation like claim charisma is from a custom game or a fan suggestion or writing exercise.

Because you said the word >>OFFICIAL<<, they can NEVER straight up just say "The wiki is wrong". They presume authority from a bunch of shitposts.

Wait until you realize half of the sources already are LLM generated diarrhea

I tend to frame questions to google from a programmer point of view - I'm carefully specific. I seem to get good results that way.

What scares me are the basic usability fails it still has. Search for a few foreign language words and it will come back with paragraphs upon paragraphs of AI output in that foreign language despite me telling Google in 15 different ways that I don't speak it, nor anything else on the Google page being in that language. How are all their products always made by and for the most narrow minded people on this planet.

> "People think that..., but also that...; It's important to notice that some people ..." where with 'people' it means just a random comment on a random website

https://en.wikipedia.org/wiki/Weasel_word

>where with 'people' it means just a random comment on a random website just because it thought it was a good contribution to the results.

Hate to break it to you, but this has been the backbone of "journalism" for the last decade.

Fishing Twitter for takes to fill the "people are saying" box...

Well, you'll be happy to know that most of American media is exactly the same way: 2 people on twitter will generate a "Americans find Widget X is bad"

> when instead it's just aggregating almost random stuff

How do you know that?

Scraping websites is literally what Google does best, stringing together information in the pattern of "some people x, other people y" requires 0 AI and could have been done since forever. I find it implausible that otherwise obviously capable models would be reduced to do something akin to just that.

Oh who cares. We are barely scratching the surface of AI. You all make it sound like it’s been around for 30 years and it sucks. It will only get better. Got to stop throwing up imaginary walls like nothing will improve.