alt Hacker NewsWhy we write our own C and C++ inference engines
Comments
I don’t think it would be surprising that people want to write their own kernels.
A big problem with the existing engines like llama or sd is that they don’t support optimal graph compilation. Usually this means about a real 2 or 3x multiplier loss relative to optimal. Cuda graphs do okay but they still leave a lot on the floor
It’s usually worth it to optimize in that context if you are willing to peer into the mechanics.
Of course that’s expensive. You need to know how to appropriately pipeline and merge your kernels.
People are critiquing whether or not AI wrote the article.
Well, it's clearly not 1-shot / low effort.
(can we agree there's a difference between low effort 1-shotting a prompt and those who use LLMs interactively?)
The one reason why I can't take a mega hardline stance against AI for writing: Not everyone speaks English as a first language.
People with awful English speaking and writing skills are using LLMs to contribute content in English. This is a good thing. They used to try their best and suffix every post with "sorry for my english".
The author's name is Ettore Di Giacinto - I put 2 and 2 together on this one.
I give a pass when non-native English speakers are just using it to write better English and keep up with the rest of us. It wasn't super obvious anyway, it wasn't interruptive.
The content is beyond that, and I more than welcome stuff like this here.
I had a similar success with Model2Vec static embedder and NER inference (both GGUF, compiled for WASM), ported to plain C from ONNX Runtime.
Wasm size from 30Mb to 300kb and 1.5x speedup. It's definitely worth it for performance or distribution size.
I did took a native c++ approach when writing a relational transformers engine (RelativeDB). My journey was pytorch -> c++ -> Triton (lang). While C++ was more performant than Triton, I couldn't afford to optimize on every gpu. I just accepted the ~15% throughput loss for my cloud service, which honestly wasn't bad for the amount of flexibility I got out of it.
But the cpp port of vllm looks great, that'd be great if you'll maintain that. I hit the same limitations with vllm.
What you get: X is the A, Y is the B.
Could this vllm port be faster to install? Im starting gpu machine multiple times a day and it takes 5 minutes to set vllm up. If Inise this port that time is minimized?
[dead]
[flagged]
[flagged]
[dead]
Ignoring the fact that this is clearly not written by a human, it's untrustworthy and the claims are dubious at best.
1. The comparison of vLLM to vLLM.cpp never shows more improvement than a handful of tokens per second. That's less than 0.05x improvement on every run, and the gap doesn't grow as concurrency increases. The comparison doesn't show any real net improvement, let alone justify the project.
2. The depth anything comparison isn't apples to apples. Of course a q8_0 quant is faster than f32. It's 4x less data to chew on.
3. This point is silly, it again fights against the argument that writing your own c++ engines are worth it. It's a bug, just fix it:
> The reason it is faster has nothing to do with writing better matmul kernels than PyTorch. Two positional embeddings, the DPT head’s UV embedding and the backbone’s bicubic position embedding, were being recomputed on every forward pass with single-threaded scalar sin, cos and bicubic loops, even though they depend only on the input geometry and are identical every call.
They argue against their own point again just after:
> For a biometric pipeline, matching the reference exactly matters more than being faster than it.
Okay, then don't rewrite it! It's not faster anyway!
4. If the argument is that the venv is large, then rewriting it in C++ doesn't seem like the answer, it seems like a lot of work and maintenance to avoid having to cull unreachable files in your venv. In a past life I maintained a simple denylist for files in node_modules. The low hanging fruit is plentiful and generally very safe.