Yes, exactly. And it's the compression that makes this happen. If it was just curve-fitting then the LLM really would act like a stochastic parrot, and it wouldn't generalize; also the model would also need to be much much bigger. As it is, the models are very big but they're still tiny compared to the dataset that they compress, so the compression process forces them to approximate a solution to the inverse problem.

And yeah, that means the base model training process is solving the inverse problem of finding an approximation for the processes that generate all human records and also anything else in the dataset with a compressible pattern to discover (weather data, etc). So it's not surprisng that some kind of world model emerges out of this.