alt Hacker NewsWhat is really impressive about this project and llama.cpp as well is that they keep finding new ways to speed up the code on the same hardware. Every time I upgrade (once a month or so) there is usually a 5 to 10% speed boost.
edit: Just to verify: I had planned to upgrade today, ran my usual benchmark prior and after and the runtime went from 4:22 to 3:59. That's a really long prompt (50K tokens) so prefill is a good chunk of that and the rest is spent on token generation.
Replies
Totally, I’ve been fiddling with local models on my machine (16GB AMD GPU + 64GB DDR4) for a few years and it’s remarkable the pace of improvement in what I can run. It has gone from complete novelty/useless to legitimately capable models running fast enough and with a big enough context to be useful running in a coding harness on a real codebase. A lot of that is down to the models themselves and the people making quants, but it feels like every time I pull llama.cpp changes and rebuild I get more performance and capability.
That's fantastic to hear. Reminds me that software improvements over the years have been roughly as effective at speeding computation as the hardware improvements have been.
What hardware are you running inference on?
I have an Intel Arc Pro B70. Serves me well! I use Qwen 3.8 27B
[dead]
While I think that llama.cpp is an outstanding project - a one-stop-shop to get almost any model to run on almost any hardware, you get a lot more speed from specialised inference engines with kernels made for your hardware.
But if you don't have time to evaluate other engines and you simply want a rock solid inference engine, then llama.cpp is one of the best ones to go to.