> I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).

Humans forget stuff all the time anyway. Would you give them the same diagnosis?

Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task.

But there's a lot of re-inforcement learning afterwards.