alt Hacker NewsBut the information in the KV cache is not words-- it's entirely latent data,and in the interior layers they KV caches don't even tend to have that much to do with words
The context gets added back to the latent vectors (which are the Q,K inputs) after every transformer layer, so they are "at least 50% words".
Also, in an N-layer transformer, there is a strong bound of N layers of uninterpretable deviation from something that is a literal token. Now sure, in a transformer with hundreds of layers that isn't saying much.
If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
I think the people who worry about interpretability (I don't) consider that a big jump.
> The context gets added back to the latent vectors (which are the Q,K inputs) after every transformer layer, so they are "at least 50% words".
Not in a modern decoder-only transformer it doesn't. The tokens are turned into embeddings which are fed into the residual layer at the front. From then on it's latents all the way down.
Each layer does a feed forward update of the state, like the euler step in ODE solver-- but what they're operating on is the evolving hidden state and not the tokens/embeddings. So there isn't any sense in which the representations must remain 'words'.
(Okay fine Qwen3.8-flash-next injects an ngram embedding at layer 2 too-- but that's it).
[Source: I've directly implemented inference engines for multiple SOTA open models on weirdo hardware-- they wouldn't have worked if I got this wrong...]
> If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
The KV state in that 1M token context is also potentially far away from any state that is observable outside of the model! (this being my point, and why I'm bothering to disagree with your correction).
Now I would agree that there being concerning hidden state is not likely and thus not a major concern, but it's not likely for the same reason that it's not likely in latent reasoning.