alt Hacker NewsCan you explain how does it work? like how is the previous K/V cache used when you switch to another model?
Source?
Replies
try-working • yesterday at 8:28 PM
edit: updated the answer above to be more qualitative instead
Can you explain how does it work? like how is the previous K/V cache used when you switch to another model?
Source?
edit: updated the answer above to be more qualitative instead
See sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).