alt Hacker NewsAmusingly constrained decoding is the same approach we use to turn an AR model into a /v1/systemone endpoint (I worked on the vLLM feature).
Tool calling is fraught with peril, especially because there's nothing that necessarily says that a models tools are fixed in place for the life of its run. In some cases, tools are added mid-run to avoid a giant prefix miss and (at least last time I checked!), the industry has not settled on OpenAI-style chat message formats for tool call updates.
I would absolutely prefer to run in a fully-constrained tool call environment, and would love if we hadn't YOLO'd the whole thing! The PTC-style tools sort of fix this by punting tool calling to JavaScript, but that's even harder to enforce, and models not trained on the style might take a turn of failed tool calls to get it right.
Replies
>some cases, tools are added mid-run to avoid a giant prefix miss
Wait, can you actually do that? The model should not exactly care where in the context the tools are, but I haven't realised it's a thing that works
What would a "fully-constrained tool call environment" be?
> avoid a giant prefix miss
I was wondering how feasible it is to modify the context, instead of just appending - e.g. to remove malformed tool calls and their error messages that would pollute the context otherwise.
Sure, you'd have to rebuild the KV cache and take a performance loss, but that might be acceptable if it happens rarely enough.