The “obviously” here is puzzling. It is certainly obvious this would be better but it is not at all obvious how to do so.
Exactly, because the whole thing is entirely based on autocompleting the next word, which is exactly why CoT prose helps getting better predictions.
Why people don't treat categorizing model activations as a science is beyond me.
There are many obvious avenues to try. This paper proposes one. I think it's likely we'll end up with some kind of auxiliary model that monitors and/or decodes the latent space of the thinking model. Certainly it is a non-trivial problem, but I don't need to know exactly how it will work in detail to believe that it is possible.