alt Hacker NewsTheir lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that aren’t true. RAG (I don’t mean embedding indices, but any information ingest such as an agent harness executing a search) are somewhat effective in covering for it, enough to make them very useful! But when it does go wrong, it’s generally the same reasons. It has a certain nature and sometimes you run afoul of it.
But I suppose it doesn’t harm its reasoning!
Replies
From your comment, I noticed a sort of pattern that is often seen when discussing LLMs. That they are these amazing things that can run so fast they trip themselves in their attempts at achieving a task. So we resort to refining the models, creating guardrails, orchestrating harness, so as to alleviate the 'hallucination' problem.
In contrast to human intelligence, there is an underlying mechanism that propels intelligent behaviour. A person is no less intelligent just because they lose sight, sound or inner voice.
They symptoms sound a lot like humans, so I don't see how it stems from their lack of self reference. Most people you need to keep them in areas they understand or they go to pieces. The lack of self reference just means every time the context clears they reset. They are systems in a permanent state of extreme amnesia.
Not sure self-reference solves the metacognition thing; an ant can pass the mirror test but probably lacks metacognition.
Though I haven't read GEB so I'm not sure how the strange loop thing ties in with either of those.
Any signal that allows the model to see what's wrong helps.
Checking code is (relatively) easy, you can use static type checks, linters, and execute it to see if it's correct.
Fact checking is harder. A RAG can only check what's in the database, so you have to know what to know beforehand.