alt Hacker NewsGreat post. I'm a first-year CS student without much math knowledge (also poor English). I'd like to check whether I got the key idea right. There are layers and each layer stores key, value pairs (normalized in this article). If I query Q then Q will be compared with every stored key "Q@A + Q@B + Q@C + ... = value". If A is the answer then Q@A will be 1, others will be interferences. D dimension only has d orthogonal directions, so if I store more keys than the dimension, I can't avoid remaining interferences. What we want is reducing them by going through function F. And this article introduces heuristic capacity results with different memory store functions (linear -> ReLU -> self attention). Is it roughly right?
self attention is the best way to suppress interferences?
Before getting too hung up on memory and seeing these as key value stores, or self-attention is suppression of interference, recognize that a lot of experts and people in these areas disagree with this analogy for fairly deep reasons (I link other resources and discussion here: https://news.ycombinator.com/item?id=50131741). It is still a useful analogy, but it is incomplete and weak in many ways when it comes to the actual details of training.
Also make sure to understand how self-attention layers are actually applied and the exact math and matrices involved. E.g. in PyTorch you are going to be doing sa(x, x, x), where x is the previous layer's output (essentially, embedded tokenization), and sa() does quite a few matrix multiplications. In cross-attention, you are doing sa(x, x, y), where y comes from a different part of the architecture.
Basically, read broadly here and don't get stuck on a single analogy / intuition for what these layers are doing. Focus on learning the actual math of the layer and the network first, and then look for analogies that help pump your intuition. But don't try too hard to force a single analogy onto the math.
EDIT: Also modern AI can provide useful insight into posts like this, I highly suggest using them when learning these things. E.g. https://chatgpt.com/share/6ad3aa3d-d110-83ea-a8c5-8653b34743...