alt Hacker NewsNeural nets can extrapolate past their training data, and there is no reason to think LLMs don’t inherit this capability.
The extent to which they are able to do this is the more interesting question!
Neural nets can extrapolate past their training data, and there is no reason to think LLMs don’t inherit this capability.
The extent to which they are able to do this is the more interesting question!
The extrapolation can also be a learned skill, especially in math. How many papers took result X, extended it to Y using known building blocks, and applied to Z.
By the way, convex hull permits extrapolating past the training data. LLM won't invent a new word that could not be defined by a sequence of known words. Just if it's meaningless and fully random/hallucinated, the new knowledge won't work with other known information blocks (breaks convexity).