logoalt Hacker News

wongarsuyesterday at 7:22 PM0 repliesview on HN

Based on the tables in part 4 it seems like some of the policies would have started overtaking LRU at larger KV-cache sizes.

And while the range of cache sizes chosen is reasonable based on the table about Moonshot's number at the start, the finding that all tested KV-cache sizes are so small that the 5-minute eviction never triggers should have been reason enough to reevaluate that choice