alt Hacker NewsThe LLM has an internal "confidence score" but that has NOTHING to do with how correct the answer is, only with how often the same words came together in training data.
E.g. getting two r's in strawberry could very well have a very high "confidence score" while a random but rare correct fact might have a very well a very low one.
In short: LLM have no concept, or even desire to produce of truth
Replies
alexwebb2 • yesterday at 10:06 AM
Huge leap there in your conclusion. Looks like you’re hand-waving away the entire phenomenon of emergent properties.
amelius • yesterday at 11:36 AM
> In short: LLM have no concept, or even desire to produce of truth
They do produce true statements most of the time, though.
➕ show 1 reply
Still, it might be interesting information to have access to, as someone running the model? Normally we are reading the output trying to build an intuition for the kinds of patterns it outputs when it's hallucinating vs creating something that happens to align with reality. Adding in this could just help with that even when it isn't always correlated to reality itself.