alt Hacker Newscurious why dont they bake the system prompt in the model itself ? Why do we pay for these tokens on every API call ?
These are just free $ for them, unnecessary bloating the context
Replies
> curious why dont they bake the system prompt in the model itself ?
Probably because if they did, they would need to retrain the model everytime they want to change the system prompt.
You don’t want to do that for anything you want to be able to vary, but they do something similar with a “soul document” for things they always want to apply.
Fully baking them in would make it expensive to update them. Caching kind of "bakes them in" (as in, removes part of the cost) while keeping it flexible.
Baking them into the model and having them apply this strongly is hard and resource intensive, as far as I am aware.
Having them in context is super easy and cheap. It is trivial to change and is 100% cacheable.
Why would it be a good idea?
That would make the model quite inflexible.
A system prompt is about guiding the behavior for the rest of the conversation.
If I'm writing an agent for financial analysis I don't want the crap that belongs to a chat-based one, or a code-oriented one.
Flexibility.
These system prompts don't affect the API, they are for the Claude consumer chat products. We aren't charged extra for them.
They're also prefix cached, so the cost to Anthropic and performance hit is greatly reduced.