alt Hacker NewsThe additional irony here is that LLMs are a tool that is likely forever damned to regurgitate knowledge of the past, with the inability to derive new information.
Replies
threethirtytwo • yesterday at 4:27 PM
This is not true at all. Just query any LLM and ask it for new information. Literally ask it to create something that doesn't exist.
It will give it to you.
It depends on what you mean, specifically on your distance metric.
If you mean nearest neighbours search like autocorrect then LLMs are extrapolative.
You can easily generate combinations not seen before. I mean you can prove this with parametric prompting.
Like "Generate a poem about {noun} in {place} in {language}" or whatever. This is a simplistic example but it doesn't take much to come up with a space that has quadrillion of possibilities. Then if you randomly sample 10 and they all seem to be "right" then you have proven it's not pure neighbour recall.
Same is true of the image generators. You can prove its not memorizing because you can generate random varients and show that the number of images realizable is more than the training data possibly contains.
If you mean on the underlying manifold of language and ideas. Its definitely interpolation, which is fundamentally a limitation of what can be done using data alone. But I know this can be expanded over iteration (I have done experiments related to this). The trick to expanding it actually running experiments/simulation on values at the boundry of the manifold. You have to run experiments on the unknown.