> but it is obviously not true for many reasons, in particular since we introduced RL.

RL or no RL, AI cannot escape the distribution it's trained on. It's just that the labs will put so much into the distribution that we won't be able to tell the difference that easily, nor will it matter for most tasks. I will probably call it AGI/ASI like everyone else when it arrives, even though it will be structurally incapable of reasoning out of distribution. There isn't a need to appeal to magic to explain OP. Just a very, very big distribution.