alt Hacker NewsIt depends on what you mean, specifically on your distance metric.
If you mean nearest neighbours search like autocorrect then LLMs are extrapolative.
You can easily generate combinations not seen before. I mean you can prove this with parametric prompting.
Like "Generate a poem about {noun} in {place} in {language}" or whatever. This is a simplistic example but it doesn't take much to come up with a space that has quadrillion of possibilities. Then if you randomly sample 10 and they all seem to be "right" then you have proven it's not pure neighbour recall.
Same is true of the image generators. You can prove its not memorizing because you can generate random varients and show that the number of images realizable is more than the training data possibly contains.
If you mean on the underlying manifold of language and ideas. Its definitely interpolation, which is fundamentally a limitation of what can be done using data alone. But I know this can be expanded over iteration (I have done experiments related to this). The trick to expanding it actually running experiments/simulation on values at the boundry of the manifold. You have to run experiments on the unknown.
It is interpolation but that is what human thinking is as well. Interpolation is so broad it can cover agi conceptually.
But I get it, the interpolation you’re talking about is limited. But I think you missed this insight: human interpolation is limited too. In the short term everything we do is simply recombination of ideas as you put it.
But that’s the short term. In the long term we do things that are much greater. But I think this is just an aggregation of small changes. Change the words in a poem 5000 times: have the LLM do the same task 5000 times. Let it pick a random word. The result is wholly original. And I think in the end this what human cognition is as well.