alt Hacker NewsGgml v1.0.0
Comments
Awesome! ggml and llama.cpp are the Linux of the AI world. Driving local inference one bit at a time.
Wishing you unlimited power and may you never see a day of low battery. Respect and Love. Keep it up.
Long time llama.cpp user (and user of other ggml based AI programs) running on my crappy AMD card that only supports vulkan and it's usable thanks to GGML and llama.cpp, great program great project.
I prefer this entry point: https://github.com/ggml-org/ggml
If you load an MoE model that slightly exceeds your GPU memory budget, llama.cpp and ggml v1.0.0 can cache less-frequently used experts in host memory rather than failing to load.
In my opinion, this is a genuine engineering product. LLM.cpp, the savior of our difficult times...
Who the heck deleted my comments, I didnt know that was a thing in Hacker news, OFC I will repost them:
-1 points by trilogic 9 hours ago | parent | context | next [–] | on: Ggml v1.0.0
> We like simplicity and aim to keep the codebase as small and as simple as possible Spawning the cli to server routing everything to http it complicates it instead of keeping it simple. This is pure telemetry or in plain english Spying on users. Rip LLama.cpp.
reply
gabucino 7 hours ago | parent | next [–]
They were already doing that by making the build process phone home to Huggingface and also download huge swathes of indecipherable JavaScript (3 of those have been backdoored a month ago and only version pinning saved downstream). reply
*
1 point by trilogic 4 hours ago | root | parent | next [–]
Yes there is much more I forgot to mention (you nailed some of it), but as I already said, everyone has a price. Although it remains a piece of engineering that empowered users certainly. Anyway we managed to eliminate all unpleasantries and have a clean CLI. I leave the server to whom may like it. Hit me up if you want to test it. reply
gabucino 4 hours ago [dead] | root | parent | next [–]
Thanks, I've also maintained my own "sanity" patchset for llama.cpp (and sent PRs), but eventually gave up in the deluge of bugs, ignored PRs (except one that I deliberately made unignoreable, but that simply got closed: https://github.com/ggml-org/llama.cpp/issues/24410) incompatible changes, and instead rolled my own engine, that's more futureproof: https://github.com/whoreson/picolm
gabucino 2 hours ago | unvote | parent | prev | next [–]
>Spawning the cli to server holy fucking shit I just checked and they really did this LOL (https://github.com/ggml-org/llama.cpp/pull/17824)
good thing i'm never touching that repo ever again, or only to steal implementations
reply
for context:
"""
Description
The main goal of ggml is to be a simple, portable, and efficient tensor library for machine learning with minimal setup.
Plain C/C++ implementation without any dependencies
Cross-platform - x86, ARM, RISC-V, LoongArch, PowerPC, s390x, and WebAssembly
SIMD-optimized kernels for x86, ARM, and RISC-V
Broad backend support - CPU, GPU, NPU, and browser
2- to 8-bit integer quantization, plus MXFP4 and NVFP4 microscaling formats
Zero memory allocations during runtime
"""Not just legacy hardware, but also ready for the immediate future, with RISC-V support.
Congrats for 1.0.0.
Congratulations on 1.0, and thanks to everyone who's worked on ggml. It's impressive how much it keeps improving on the same hardware. I'm curious whether 1.0 also comes with a stability commitment for the public C API. I've been looking at Python bindings to embed llama.cpp in-process, and knowing the API is stable until a 2.0 would make building on top of it much easier.
[flagged]
I've been running llama.cpp on a plain CPU-only box for local work for over a year now, and the thing that keeps surprising me is how often an upgrade is just free performance. Same model, same hardware, noticeably faster prompt processing — jacquesm's 4:22 to 3:59 matches what I've seen on my own machine. The other underrated part is that the codebase is small enough to actually read. When something is slow I can usually find the hot loop myself instead of treating the inference engine as a black box, which is not something I can say about most of the GPU stacks I've used.
[flagged]
[flagged]
Imagine naming a library after yourself.
> GGML is a C library for machine learning (ML) - the "GG" refers to the initials of its originator (Georgi Gerganov).
https://github.com/rustformers/llm/blob/main/crates/ggml/REA...
[flagged]
What is really impressive about this project and llama.cpp as well is that they keep finding new ways to speed up the code on the same hardware. Every time I upgrade (once a month or so) there is usually a 5 to 10% speed boost.
edit: Just to verify: I had planned to upgrade today, ran my usual benchmark prior and after and the runtime went from 4:22 to 3:59. That's a really long prompt (50K tokens) so prefill is a good chunk of that and the rest is spent on token generation.