alt Hacker NewsI'd approach this distinction differently. Prediction from compression is valid within the distribution of the compressed data. Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)
> Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)
This is a great experiment. Do some world building offline about Smergle the Keen finding the Hoopla of Kynashiusous. Then ask ChatGPT what reagents would have been more effective in breaking through the borantiastic door than squeeble stomach juices.
Of course we know the answer - it will either drop into a 'I have only been trained on data before blah blah blah' or 'I cannot talk about current world leaders' or 'I have not been trained on your hackneyed and unpublished romantasy saga, why don't you tell me about it?'
On the other hand, if it is aware of the general concepts of your concept, and has bridging concepts for related topics it has been trained upon, it might try to apply the general concepts to a specific but different scenario.