alt Hacker News> NeMo Switchyard, an open source library for smart routing
> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job
How do routers like this handle prompt caching when you send the second request?
Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
Replies
https://github.com/NVIDIA-NeMo/Switchyard#routing-strategies
Looks like your great question doesn’t have an answer, but looking at the routing strategies things get even more confused, since the proposed ones tend to rely on extra llm calls to determine which model to pick.
The nice thing is that it makes sense for specific setups, less conversation oriented.
As an example, you need to classify batches of data, and have many fine tuned models. Or you need to do speed to text and need to pick which whisper to use.
You can write your own strategy, in that case an harness with subagents would be able to leverage this, picking the right model and then keeping its session sticky, but overall the lack of concern for caching points towards use cases where you do not gain much from it.
The repo is probably a better entrypoint to it, bit more concise description than the press releases: https://github.com/NVIDIA-NeMo/Switchyard (Notably: "Experimental software. Not for production use."). Unclear if they actually want you to deploy it or not, press release says yes, README says no, do with that what you will.
Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
Caching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed.
I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
I've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn).
But yeah I'm skeptical all this overhead is worth it.