alt Hacker NewsI find some aspects of your writeup to be... quite annoying if I'm honest. The writing is good, but you fail to define several of your terms early on.
What is d_k - I assume it's the dimension of the k vector. Similarly, I assume d_v is the dimension of the v vector. What is the "i" suffix for here at this early stage? It doesn't need to be there, it's just confusing.
You say the keys are "used for lookup based on similarity". What does this mean? Similarity to what? The existing keys? Something else?
Your first example then introduces the identifier "M", which seems to be the number of memories. Again, this is not defined. It speaks again of similarities, and introduces "weights". But the matrix Mxd_k is a matrix made of all your keys, which aren't "weights" - or at least this term is new. The term "tokens" makes an appearance around this time also - something else that needs defining.
I think you need to go through the text of this article, with an eye to someone who is reading it for the first time. Define every term you use, remove unnecessary notation - and also perhaps explain at the start why one would be interested in a "memory store" of this nature.
There is a place for something that really explains what's going on here. I've yet to find it.
Replies
I found many of the things are clearly defined after the sentence „Here are the terms we’ll use throughout.“ right at the beginning (e.g. M, d_k, d_v).
Instead of being condescending and annoying yourself, did you even try to reference other texts or resources to try to figure those things out? This is a freshman's complaint
Agreed 100%. Mathematicians SUCK at writing. No matter how hard they try, they'll gloss over important non-trivial definitions. And I think this author has been trying... but seems to at the same time ignore the negative feedback like yet another academia snob.