alt Hacker NewsFor anyone here who understands this stuff, help me out because the article dropped me at the end when drawing the connection from memory to LLMs.
For an LLM, the keys are the tokens in your context, and the values are … what exactly?
Replies
D-Machine • today at 4:44 PM
See my comments and links below, you are not wrong, IMO this is not actually a great analogy for understanding LLMs (other than as an explanation for why a trained one can so efficiently compress such a large amount of information).
In an attention layer, the query, key, and value are all computed at inference time using weight matrices that are determined as a result of the training. If X is the token representations entering the attention layer, then each of Q, K, and V would be determined by multiplying X by the corresponding learned weight matrix for queries, keys, and values:
Q = XW_q
K = XW_k
V = XW_v
From there, you can think of K and V like keys and values in a Python dictionary, and Q as the key you pass (e.g. d[Q] in Python to return some value V). The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.