alt Hacker NewsCan you expand on this? I can't imagine this is holding up an entire machine or materially impacting capacity. Isn't it common to have a tiered cache and to ultimately evict (or move to another tier) if there's an active inference request and no available capacity elsewhere?
Yes, however if a tool like this commoditizes a way to stay "higher" in the cache hierarchy to avoid getting dropped, then for the same total capacity, Anthropic have to become stricter about retention.