alt Hacker NewsI think the Kimi thing is super cool, especially that they have so many RNN/linear attention layers (3x more than they have full attention). I haven't yet tried it though. It seems like it would be extremely reasonable for long context tasks and I guess this fits the times.
I suspect that the reason it has so many parameters is the same reason that compute optimal xLSTMs have some many parameters, and the success of this model makes me a bit unhappy that we haven't gotten an xLSTM-style model of huge size developed in Europe.
Obviously these guys are very pragmatic, they're probably not committed to anything other than what works on their internal evaluations, so they still have ordinary attention layers in the model and so on, and one can't be guaranteed that the people who come up with a good model then do the engineering in an ideal way, but I still think the success of Kimi shows what could have been if we had enough big supercomputers for LLM training and made them available to the right people-- because this is basically Hochreiter's thing. It's RNNs, or well, mostly RNNs.
Replies
I talked with Hochreiter about 1.5y ago they should go for it and properly scale xLSTM to LLMs, not just playing around with few B params for experiments. They felt like they are too late to the game to get investment and traction. So much for European ambitions I guess...
Is it an RNN or a state space one, like qwen or there was an arch called mamba if I remember correctly. Or it might be closer to RWKV.
Edit: I checked and it is similar to linear delta net uses by qeen.
Wasn't one of the original intents of the transformer architecture to get rid of RNN because they cannot be parallelized? Not an expert, just read a few papers several years ago.