alt Hacker NewsWhile I think that llama.cpp is an outstanding project - a one-stop-shop to get almost any model to run on almost any hardware, you get a lot more speed from specialised inference engines with kernels made for your hardware.
But if you don't have time to evaluate other engines and you simply want a rock solid inference engine, then llama.cpp is one of the best ones to go to.
Replies
I run vllm and llama.cpp; llama.cpp is pretty hackable by a local agent and is fast to start up, while vllm takes and age to build its optimized cuda graph.
The result is I run my big model (glm 5.3 flash) on llama.cpp and my small utility model (Qwen3 vl 8b) on vllm. Qwen often has better vision than glm (despite the tiny size) and can respond to tool calls from glm and importantly it doesn't use gobs of prefill+kv and is super fast.
For this project it's more about achieving the 'train once, run everywhere' paradigm that Java had.
> you get a lot more speed from specialised inference engines with kernels made for your hardware.
Of course you will. But that's not the point. The point is that Llama runs just about every model on just about every hardware. You can always improve on any metric by dropping that constraint. But there are a lot of such metrics:
- ease of use
- range of hardware
- accuracy of results
- range of models
Probably others. Llama.cpp may not be best at any one of those. But across all of them it has been consistently good. They've shown they're willing to stay the course as well. This is a foundation level project much like Linux or Apache.