alt Hacker NewsAre there any reproducible hallucinations on any of the currently available OAI/Anthropic models? I’m not aware of any.
And even if they are related - if Opus 4.8 always has a 1:100 chance of a specific hallucination - then running the same model twice does indeed dramatically reduce the odds of an error in the final output.
Replies
If simply running things thrice-over was enough to stop "hallucinations" (and not incur other problems) we wouldn't be here talking about it today, it'd have been "solved" months or years ago.
Constant hallucinations. OpenAI:s latest on max settings. If you are to naively feed say, a short corpus of text to turn it into a parallel corpus in a few different languages, the original text gets subtly mangled and no longer matches the original. Say you have several hundred annotated sentences. Without hand-coding some regex to make sure that each sentence in the source column occurs in the original corpus you’re bound to get hallucinated sentences with an error rate that exceeds 1:100. Whatever you use as the output, JSON or XML, you will end up with columns that just repeat the original instead of translating it, especially for languages that are very close to each other or represent the same language.
Yes, LLMs can be SOTA for NLP, but you’re going to have to use them to write software or workflows that are more deterministic.
Even if your conditions are granted, different LLMs can easily consistently hold divergent "worldviews". LLM A might consistently state that fact P is true (i.e. it will never hallucinate a string that states or implies ¬P), while LLM B consistently states that fact ¬P is true. You don't have a way out of this with these two, and adding a third LLM is an additional confounding factor, not a solution. You can't settle questions of fact by survey, you need to test or use reliable sources.