logoalt Hacker News

Gracanatoday at 2:47 AM1 replyview on HN

What’s that about the template breaking the kv cache?


Replies

xlayntoday at 2:51 AM

this is my understanding, the default template keeps the thinking part but only for the last message, so the harness has to play along with the template and strip and add to keep the conversation matching what's there on the llama.cpp cache, but if the harness sends the thinking in every turn, then you break what llama.cpp expects, the conversation doesn't match anymore what you have on cache and it reprocesses again the whole conversation

show 1 reply