I know this is a popular position and it makes sense at face value when you think of LLMs as autocomplete systems. But it’s genuinely wrong.

Relevant reading is most notably anthropic’s research on the J-space. LLMs will plan ahead of time helped with CoT, get to a plan and “store” it in j-space, and execute on that plan which means they can in fact “backtrack” and give you reasoning on why they did something, because it IS part of their state.