logoalt Hacker News

foo12bartoday at 11:23 AM0 repliesview on HN

If we're talking about running it locally, what about passing a partial response as part of the input?

Prompt part: "What is better, toast or bread?"

Incomplete answer part: "The answer to this question is "

and then have the LLM finish the answer. I did this with subtitle translation using llama.cpp (with Python) and had great success. Just past 5 already translated subtitles as the incomplete answer, and the LLM infallibly just continues to translate. No markdown, and usually no talkback if the subtitles contain nasty subjects like bioweapons or nuclear stuff. It just works.