alt Hacker Newsif it puts a high confidence value on a wrong answer, thats still hallucinating, no?
llm hallucinations are high probability tokens that are incorrect vs the real world
Replies
No, I don't believe so. Hallucinations are not "high probability" in a real sense. They are an artifact of the random walk the inference algorithm takes, which causes it to latch on to and chase attractors in the noise. This random walk behavior is necessary for chat interfaces to be useful, but are less critical to typed output predictors. I'm guessing they found some optimization that is possible if you give up caring about chat.
Yes, there is no magic sauce here that makes stochastic output binary if that’s what people are looking for.
What we would want to see if a confidence value that is in line with the actual correctness. If the value is 0.9 for 1000 different answers, then approximately 900 of those answers should be correct.
Correct, they have not made a universal all-knowing omniscient oracle, which is what would be required for "can't hallucinate".