alt Hacker NewsOld news by now, but you might not be aware that llama-server can do multi-model for a while now,
Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI)
Then, any api client can just select a model and the system does the right thing.
It's great software. It just works.
__
You just need to ignore the cargo culting commandline options on social media. But you should be listening to the devs.
Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.
Replies
Is that with vanilla llama.cpp or with the third-party llama-swap manager? Last time I checked llama-swap was still the go-to solution, although I admit I haven't looked into it further.
> (Optimized by you through testing. Not that AI)
Why not optimized by AI through testing ? Give it a test set to work on and let it loose.
[dead]
Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.