alt Hacker NewsWhy does it need to be the average? It seems to me more like it models the manifold of human knowledge. However we often query for the average, because that is often good enough and gives us quick results, but there is nothing fundamentally preventing us from sending AI into the deep end of under-explored territory and perhaps coming back with something new. It is ultimately the exploration vs exploitation trade off.
> but there is nothing fundamentally preventing us from sending AI into the deep end of under-explored territory and perhaps coming back with something new
What's stopping us is that AI works by manipulating tokens and language but has no connection to reality as it exists. Einstein famously conceived of special relativity by imagining what it would be like to fly alongside a light wave[1]. This is a process that integrates spatial reasoning and imagination informed by living in the real universe where you can see objects moving or waves propagating in a pond. The language only comes later as a means of communicating these intuitions to others.
[1] https://sites.pitt.edu/~jdnorton/Goodies/Chasing_the_light/