I run vllm and llama.cpp; llama.cpp is pretty hackable by a local agent and is fast to start up, while vllm takes and age to build its optimized cuda graph.

The result is I run my big model (glm 5.3 flash) on llama.cpp and my small utility model (Qwen3 vl 8b) on vllm. Qwen often has better vision than glm (despite the tiny size) and can respond to tool calls from glm and importantly it doesn't use gobs of prefill+kv and is super fast.