How can you be hitting cache on what I think are novel LLM prompts …
multi turn sessions, they are typically in the high 90% hit rate across all providers without doing much of anything
The prompt is only novel the first time it’s sent. Then as it’s sent repeatedly as part of the previous context it’s cached.
Deepseek afaik has a novel architecture that is somewhat forgiving of cache shifts. I’ve been getting +90% cache hits with Zed’s agent and it’s not doing anything special regarding caching afaik.
Not them but my understanding is that the harness will send a simple 'heartbeat' message to keep the cache 'warm', (see prefix caching: https://handbook.modular.com/inference-optimization/prefix-c... ) which can then be edited/changed, which does cause the user to incur a fee, but its much less than the amount they'd pay on a no-cache hit request.