alt Hacker News> The worst part is that everyone who's decided to willingly lobotomize themselves is going to have to come to the realization that these things are full of shit. It'll have to happen one by one, and nobody else can make it happen for them.
A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would.
Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.
Replies
> LLMs are very convincing and persuasive.
The goal of LLM's, as they are marketed now, is to drive engagement and stickyness of products. A wrong answer is brushed off with a "Hey, you're right, let's try that again" - a response purposely designed to maximise the friendliness of the system and minimise the sting of a wrong answer. The fact that an LLM will not respond to the same question in the same way twice (i.e. the 'temperature' ) is because increased accuracy will not drive engagement and, therefore, increased accuracy cannot be allowed to get in the way of engagement.
> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.
Psychics are still in business. Although to be fair they probably don't have as much revenue. I think people really enjoy being told how smart and insightful they are, and how much they've really cut to the crux of the issue. This isn't the whole thing, but I think it counts for a lot.
> Surely if the machine you go to for answers regularly makes things up you would just stop using it?
Steve Yegge likened LLMs to slot machines. The human brain is very vulnerable to random reward systems. If you get an hallucination, just pull the lever once more.
> Surely if the machine you go to for answers regularly makes things up you would just stop using it?
We live in hope that people stop doing stupid things and are constantly disappointed.
> Surely if the machine you go to for answers regularly makes things up you would just stop using it? Perhaps people have to be burned by something really bad personally before they realise the limitations? LLMs are very convincing and persuasive.
The last sentence reflects a lot of my feelings on the first question. LLMs have a sort of weaponized take on the ELIZA Effect. The better their memory the better they are at playing to human social desires to be listened to in an active conversation. At some point it stops mattering if the answers are right when the answers feel right, but really, like ELIZA back in the day, so much of what makes it seem special is just reflecting your own writing back at you in a convincing and persuasive way.
> Surely if the machine you go to for answers regularly makes things up you would just stop using it?
People still respond to ads and political speeches.
> A fascinating dichotomy has become apparent between those who trust LLM output and those who don’t and don’t understand why you would.
I feel like the most pragmatic perspective is "trust but verify."
This is why they're so effective at coding: you can run the code yourself (or the test suite) to verify that it actually does what it's supposed to.
And maybe these people finding hallucinated results on Rachel's site are doing verification too.
>Surely if the machine you go to for answers regularly makes things up you would just stop using it?
Not necessarily, namely because P != NP. Verifying the correctness of a solution is faster than solving it. Thus a system that outputs 99% incorrect solutions and 1% correct solutions can still be incredibly useful.
even worse when managers, while sharing their screen, read an assertion from Gemini and take as fact. puts subordinates in a position where theyre responsible for challenging the assertion (if warranted) and, in a way, challenge their manager's decision making
not saying anything new. easy enough to frame it like any other assistant and check references
I have to admit since VSCode seems to be regularly re-enabling the Cocaine Parrot Autocomplete my views on LLMs and coding has softened a little.
I'll temper that slightly by saying it's mostly out of morbid curiosity because the things that the Dreaming Piracy Robot comes up with are frequently wildly incorrect code, but it's interesting to think about how it might have got there.
And then I think, well, maybe Special Needs Wintermute has a point. Maybe there's a different way to think about it that I've missed.
And then I just change it back to what I wanted in the first place.
I bet there will be a very interesting generational divide between the kids that were born before or after about 2010; old enough to have some critical thinking facilities at the dawn of ChatGPT when it was still noticeably dumb.
Even if LLMs lied 30% of the time, they would still be about as useful as they currently are for me.
When I ask for their input, it's always for a situation where I'm capable of judging if their input is useful or not.
In all situations I use them, it doesn't matter if they're correct at all. I'm asking for ideas, alternatives, links for blogs or articles. I talk things out with them...
I don't think we should ever "trust" LLMs. This seems like the wrong usecase for them.
Nowadays not just big orgs are falling into the delusion, but also big people.
When Linus posted that AIs and vibecoding were here to stay and declared resistance to it as harmful, I stopped to consider whether I was wrong, but it has made me realize that in retrospect Linus Torvalds and Linux itself aren't actually the holy grail of computing. I didn't feel that way with Richard Dawkins, its not like falling for an AI psychosis retroactively made me question The Selfish Gene, but now I'm looking at linux and the theory that it's a clusterfuck is gaining so much traction, especially after copy.fail and ensuing rustification, I see so much clearly now. It was never about linux, UNIX sure, POSIX, yeah, GNU fucking aye, kernel? Ok whatever, drivers and scheduler with a gajillion lines of code I guess.
Making things up is only really a common issue on the non-thinking models which nobody should be using. The regular chatbots are Autogooglers and are very useful for research. This is just not a good argument anymore.
Edit: Guys, why are we downvoting this? Does no one use like ChatGPT or Claude and understand how it works? Do you all think its regularly hallucinating links still? Is everyone on HN using like free signed out accounts or something? What year is it?
I have a distinct line between when I'm willing to believe an LLM's output and when I'm not: whether I would believe the same thing from an anonymous Internet forum post or a blogger I don't know. Those posts are not unlikely to be misinformed, biased, lies, or otherwise untrustworthy. And yet, I spent plenty of years honing a sense of when they were good enough for certain things.