Let's not forget about Yann LeCun's current area of research that's completely different from LLMs: Joint Embedding Predictive Architecture (JEPA)

If he gets that style to be more efficient (they're already competitive) it'll completely kill off LLMs

https://openreview.net/pdf?id=BZ5a1r-kVsf