alt Hacker NewsI am not sure you're correct. Yes, from the physical perspective of LLMs design, the above is correct. But they are also systems that evolved to do things through training; so you cannot discount the possibility they can emulate building of new abstractions in another level.
LLMs already have the concept of "reference" - i.e. I can say a name of something, for example of a function, and they understand that the properties associated with the name itself are not relevant, but rather whatever the name does refer to.
But this is pretty much what you need to build new abstractions. You need to name a paragraph with the definition (explanation) of a concept, and be able to recall the paragraph by the name in order to do inference.
So with references, they can get around the context size limitations, like humans can. It's slower than learning things "by heart", but I think LLMs are perfectly capable of that; and the evidence is, they are able to navigate codebases larger than their context, which often contain many new abstractions.
I can say what I was getting at quite definitively. An LLM is, to an approximation, a pure function. No side effects, no state changes. No neuroplasticity. No working memory. It really is just predicting next tokens based on previous tokens. So the only way to change its output for a given input is to change the input - that is, pack different information into the context window.
Some nondeterminism from the probabilistic token sampling and distributed inference, of course. But sometimes randomly choosing the second most likely next token isn’t going to manifest a cognitive feedback loop.