alt Hacker NewsIn an attention layer, the query, key, and value are all computed at inference time using weight matrices that are determined as a result of the training. If X is the token representations entering the attention layer, then each of Q, K, and V would be determined by multiplying X by the corresponding learned weight matrix for queries, keys, and values:
Q = XW_q
K = XW_k
V = XW_v
From there, you can think of K and V like keys and values in a Python dictionary, and Q as the key you pass (e.g. d[Q] in Python to return some value V). The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
Replies
To clarify slightly, K and V are used to construct the dictionary, and then Q is the lookup key D[Q] as described (with the lookup being based on similarity rather than exact hash as it would be in an actual dictionary).
> The difference in an attention layer is that the query is matched to each key using a dot product, and rather than getting a single value back you get a weighted sum of values.
I understand you meant to compare it to dicts, but this is more like (the only?) similarity to outer product memory. Worth calling out since GP is confused about this connection specifically.