logoalt Hacker News

snailmailman • today at 6:31 PM • 1 reply • view on HN

Anecdotally, the percentage is way higher than 10% for some kinds of searches. I don't always use google, but i've tried it dozens of times the past few weeks to test their AI Summary / AI Overview, and it has yet to get anything correct. I've had to use other search engines or AIs literally every time.

mistakes vary from "obviously wrong" to "looks vaguely correct, but the source states something different" to "looks plausible, but the cited page is completely irrelevant, the AI clearly made this up" Even for straight fact lookups, I've seen it cite pages and claim one number, and then i click through and the source matches the vague shape of the answer but has a different number.

Its just constant hallucinations. "thats a common problem, just change this <setting that does not exist> or set <option that does not exist>". Full gemini seems more capable, but whatever google is throwing at the top of their free searches is awful IME.


Replies

smallerize • today at 8:15 PM

I'm guessing they're only counting one failure mode. But I think it's the one that would surprise most boosters. Even if it found a relevant page, and the information is fed into the model's context right there, the output is still completely backwards a fair fraction of the time.