alt Hacker NewsNote that this assumes a single chat (single cached context). As soon as you want to use it for several tasks in parallel or in frequent alternation, hardware requirements increase or performance drops.
Note that this assumes a single chat (single cached context). As soon as you want to use it for several tasks in parallel or in frequent alternation, hardware requirements increase or performance drops.
I mean I haven't checked but I'd guess it'd be possible to save a kv cache to an ssd or ram and then pull it back into vram when needed.