The simplest way to "prove" this is to show that a model can emit an image that isn't part of its training set. I'm sure you've seen that a million times already - what is a "fake" if all "fakes" are regurgitation of a real image?

That means models are capable of outputs that were never their inputs.

It's easy to see why this happens if you think of training as selecting a target against which the model will be scored, and then randomly mutating the model until it is close-enough to the target. There is no guarantee the random mutations produce something identical to the target for ANY of the training data. A model might finish training at at average delta of 10%, and it's possible that means that model is incapable of exactly producing anything in its training data (it could be 90% accurate for any prompt, or it could be 100% accurate for some and 80% for others, or...).

In addition, prompts that were never present in the training data might produce horrific or random outputs, because those were never even scored during training so they might get something right-ish or something entirely full of garbage data. No telling.

And this is all ON TOP OF the nondeterminism in inference. Everything I said above would still be totally true even if inference were determinstically done.