logoalt Hacker News

soerxpsotoday at 7:36 PM1 replyview on HN

My understanding of caching with most models/providers is that a prefix substring of the context has to be reused for a cache hit, but not necessarily the whole entire context window. So if you prune tool calls from the history, you're going to get one cache miss on the newly-pruned history, and then you're going to be getting cache hits on every subsequent turn, with a lower number of input tokens. If you prune subsequent tool calls after that, you would still get a cache hit for the already-pruned portion of the context, just not the full context.


Replies

__natty__today at 8:05 PM

So it makes sense to first send stable prompt, reasoning and files content, tool calls summary and actual tool calls at the very end?

show 1 reply