alt Hacker NewsYes, but we are really talking about LRMs not LLMs. They can emulate working memory so well that many people even recommend using bunch of .md files (as a sort of out-of-context notes) instead of RAG based on vector DBs. They can maintain context, by compressing it and writing out memories quite autonomously.
So yes, even though not updating their weights, they can learn plenty new things and concepts, and build new models of the world.
But now you’ve looped back around to my original point: a large context window is essential to making it work well. Those *.md files won’t do anything if you don’t feed them into the LLM at inference time.
(edit: That said i do agree with the idea of designing systems so that such a large window isn’t needed. But that seems a bit self defeating if the goal is massive development acceleration with agentic coding. These models seem to be pretty bad at engineering for simplicity. And protecting system design with close human oversight would mean we are still limited by the speed at which humans can read and understand code.)