Can't LLMs just "degrade" into decision APIs whenever possible/necessary?
Why does this need to be a separate thing?
You could instruct them to behave like one but they’ll still eat a lot of tokens, probably more, even though you’ll save a small amount on the output. It’s a separate thing because of significant token savings.
They generally use smaller, more specialised models which are substantially cheaper for similar performance e.g. https://strandsagents.com/blog/amazon-saves-millions-replaci...