alt Hacker NewsIt’s a shitload of data, and it only works if all the tokens are 100% identical, i.e. all the attention values are exactly the same.
Typically it’s cached for about 5 minutes, you can pay extra for longer caches.
It’s a shitload of data, and it only works if all the tokens are 100% identical, i.e. all the attention values are exactly the same.
Typically it’s cached for about 5 minutes, you can pay extra for longer caches.
If I have a conversation with claude then come back 30 minutes later to resume the conversation, the KV values for that prefill prefix are going to be exactly the same. That's the whole point of this caching in the first place.
If you're willing to incur a latency penalty on a "cold resume" (which is fine for most use-cases), why couldn't they just move it to disk. The size of the KV cache should scale on the order of something like (context_length * n_layers * residual_length). I think for a standard V3-MoE model at 1M token length, this should be on the order of 100G at FP16? And you can surely play tricks with KV compression (e.g. the recent TurboQuant paper). It doesn't seem like an outrageous amount of data to put onto cheap scratch HDD (and it doesn't grow indefinitely since really old conversations can be discarded).