More like: the output of genAI are probably the average knowledge of all the human training data.

Why does it need to be the average? It seems to me more like it models the manifold of human knowledge. However we often query for the average, because that is often good enough and gives us quick results, but there is nothing fundamentally preventing us from sending AI into the deep end of under-explored territory and perhaps coming back with something new. It is ultimately the exploration vs exploitation trade off.

Not quite. Large amounts of data going into these models has already been curated, otherwise you would get a tremendous amount of wrong answers for even the most basic questions.

It is the average of sub sets of humans, which can be vastly better than just entire population.

Example it can easily do moderately advanced calculus, that is way better than the average human.

While true in some sense, it does have more knowledge than the average person.

That is not true. AI can synthesize information, which is the essence of intelligence. And since it knows more than everyone, it is more intelligent too. What they lack is the ability to create information.