That is ephemeral context. Not permanent learning.
You might ask “why not make the context larger ?” Well, that just spreads out the “attention” focus wider and wider, and you get the “lost in the middle” effect as details get blurred over and not surfaced to results.
You might ask “why not make the context longer-lasting ?” Well, that means the signal-to-noise ratio drops as more and more information accrues over time that isn’t germane to the problem at hand.
As I said, LLMs as we know them today are effectively lookup devices, very very clever ones, but they lack comprehension, and without that there is no value judgement on what is useful (and conversely what is not.) They lack the ability to forget. They lack the ability to create new permanent storage of information. The gap is not some small details, it’s the difference between single-cell and multi-cellular life.
Again, just tell me the capability rather than speculate about "spreading attention and focus" or "lookup machines". Show how humans can do X, but LLMs can't. I can't reason well (and haven't heard anyone else do so convincingly) about what a sufficiently advanced lookup machine can't do compared to a human brain.