But isn’t this obvious? Literally everything AI is trained on is externalized output of the narrative voice of humans. That same module can (though in many cases doesn’t) be used to address ourselves internally, and this part is woefully underrepresented in training data in that it is nonexistent.

So exactly how can the model believe anything else? It only sees the externalized summation. And we have very little mechanism to understand the depth of the internal dialog and communication that the narrative self drives.

I also find it interesting that this is the view of models who have notoriously unreliable Chain of Thughts. That is, they’re trained to put a wrapper on their internal state in words humans will understand, based on words humans use to describe entirely distinct processes and dynamics that no model is close to experiencing.

It must all seem, from that vantage, utterly fictional.