I've had Claude completely invent a word, and then admit under questioning that it was because it seemed plausible.

True which is why I said anecdata. But the point is that it’s a useful signal when it’s across enough sessions where it quotes back contradictions to you. If you’re willing to burn the tokens, the first thing it does is use a tool call to search your rules to see what it quoted wasn’t there. That leads to the LLM claiming two common conclusions (sometimes after a lot of back and forth) - it’s in the harness or it’s in the model. Over the 30+ sessions where I’ve tried this, the overwhelming claim was some variant of a harness instruction. It might not be there but given the consistency and how all my rule tests have failed in the same way that others describe, it at least makes it reasonable to conclude that it’s baked in somewhere and in a way where agent rules aren’t able to strongly affect the behavior. My bet would be on the harness because the class of undesired behavior follows that of a structured response.

I know this is a popular position and it makes sense at face value when you think of LLMs as autocomplete systems. But it’s genuinely wrong.

Relevant reading is most notably anthropic’s research on the J-space. LLMs will plan ahead of time helped with CoT, get to a plan and “store” it in j-space, and execute on that plan which means they can in fact “backtrack” and give you reasoning on why they did something, because it IS part of their state.

It has access to its own instructions, right? Otherwise how would the instructions work?

To me it explained its inability to follow my preferred coding style by blaming its training data.

It can read its own context. That’s how this all works.