alt Hacker NewsDefinitely, LLMs are highly ineficient now.
The diffusion models are interesting, but those also seem hacky.
I think the next form of AIs will be simpler and more abstract.
The building blocks of our brain don't have the notion of a "token" embed into them, it's lower level that that.
I think first step is to find a better way to represent information.
LLMs shouldn't "compute" stuff using language tokens, but some other, more efficient logical mechanisms. LLMs should first "feel" the solution, reason internally in that optimised space, then, only when interacting with a human should it convert all that into actual tokens/language.
The problem with that is we don't have any kind of training data in that abstract sense, maybe we could use RL to figure that out but current RL techniques are too slow and prone to breakage that anybody trying to use them to train a big enough general model (LLM, diffusion, world model, etc) will either fail or have to make a very very big investment.
The other option is maybe hook up humans to EEG or the likes and map their brains while they solve different kinds of problem, or just see and feel the world around them