alt Hacker NewsAre you modeling the actual API costs? If you can cut round trips at the end of a long context window you can end up reducing usage even if the total token count is the same.
Are you modeling the actual API costs? If you can cut round trips at the end of a long context window you can end up reducing usage even if the total token count is the same.
That's a great point, thanks! I will include an eval for this case.