Indeed. Who'd have thought that mere statistics could do that? Hat's off to them - they made the darn thing work.

There's pretty interesting (I think) research in how effectively it's possible to scale down LLMs. If you're interested, take a look at the BabyLM conferences:

https://babylm.github.io/

(I have no connection to them.) The work there demonstrates that gap between what we can do now with human-relevant amounts of training text, and what remains to be discovered. Quite a bit, it looks like.