This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates.

Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase.

Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.

> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards

LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.

Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.

Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.

Take yourself out of that loop so you only find out once it's sure.

I've gone in loops where it'll tell me twenty times in a row that it's definitely found the root cause of a bug, when in fact it definitely did not.

A lot of weirdness in their style is the fact they think in-band.

Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare.

To write an honest answer, it says, "Here's the honest answer", and proceeds from there.

If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker".

This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext.

You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.

LLM aren't logic machines. It's concepts of "root cause" and "faster" are purely linguistic, including all the internal conflicts of those loose definitions.

In that sense an LLM wouldn't even think it "lied" being confidently incorrect, in human terms it's just a "whoopsie". Apparently we want exactly that.