This is one of the most well written blog posts I've seen in a while. Thanks

Thank you for putting that excellent article together

The concepts of fidelity/recall and the images/animation at the top reminded me of this small playful project that touches upon these topics: https://nicobrenner.github.io/lossy-git/

https://www.nature.com/articles/s41586-024-08548-w

and others. Hippocampal sparse coding/orthoganalization/pattern separation and completion/etc may be interesting search terms for you, if you are unfamiliar. If you do happen to be a hippocampal nut like I used to be during grad school, I salute you

No, after many years, I still hate Twitter/X.

Please write a proper Blog post.

I find some aspects of your writeup to be... quite annoying if I'm honest. The writing is good, but you fail to define several of your terms early on.

What is d_k - I assume it's the dimension of the k vector. Similarly, I assume d_v is the dimension of the v vector. What is the "i" suffix for here at this early stage? It doesn't need to be there, it's just confusing.

You say the keys are "used for lookup based on similarity". What does this mean? Similarity to what? The existing keys? Something else?

Your first example then introduces the identifier "M", which seems to be the number of memories. Again, this is not defined. It speaks again of similarities, and introduces "weights". But the matrix Mxd_k is a matrix made of all your keys, which aren't "weights" - or at least this term is new. The term "tokens" makes an appearance around this time also - something else that needs defining.

I think you need to go through the text of this article, with an eye to someone who is reading it for the first time. Define every term you use, remove unnecessary notation - and also perhaps explain at the start why one would be interested in a "memory store" of this nature.

There is a place for something that really explains what's going on here. I've yet to find it.

Fantastic blog post.

Curious to hear more about the stack for your blog? Thanks

Really awesome write-up, and the kea is adorable!