alt Hacker News"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."
It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?
Replies
itkovian_ • today at 8:51 PM
Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!).
And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.