alt Hacker NewsI personally think of this like sorting algorithms. Quick sort does the same thing bubble sort does so why do we need quick sort? Pushing for efficiency drives innovation. It does this for many reasons but a big one is that putting a cap on a resource forces you to consider the others available and often you find that all it took was a little effort and suddenly the alternate path that looked a little worse is actually better than you realized.
This has a lot to do with how MCTS works BTW. The current best path is often only the current best path because a lot of investment has been sunk into it. If you were to put equal resources into a different path you may find that it was actually far better. It is just that the early rollouts favored the other 'best path' so you sunk a lot of resources into that one. We are very early in our exploration of LLM architecture. I highly doubt we are anywhere near the best path right now.
Definitely, LLMs are highly ineficient now.
The diffusion models are interesting, but those also seem hacky.
I think the next form of AIs will be simpler and more abstract.
The building blocks of our brain don't have the notion of a "token" embed into them, it's lower level that that.
I think first step is to find a better way to represent information.
LLMs shouldn't "compute" stuff using language tokens, but some other, more efficient logical mechanisms. LLMs should first "feel" the solution, reason internally in that optimised space, then, only when interacting with a human should it convert all that into actual tokens/language.