Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA

README is in my opinion (author here) the most interesting - I wrote it to help others build useful mental model to be able to recreate the project yourself, without need to even read my code

I feel like I learned twice as much in 10 minutes reading this than I did reading LLM for Dummies. Thank you

The lesson-style README is a great approach. Breaking down LLM inference into digestible steps makes the codebase approachable even for people who haven't touched CUDA before.

Very nice job on read me.

>>Physically, LLM is a file which contains a lot of float numbers.

aka atoms of the LLM.

Thanks for sharing this. As someone currently researching LLMs, I'm sure I'll be referencing this quite a bit going forward.

interesting!

Looks interesting, it reminds me of the first llama.cpp, but better documented.

I love the documentation formatted in lessons. I can't wait to read through it.

Wanted to add that the author has an amazing blog with lots of interesting papers: https://jedrzej.maczan.pl/

I am looking at a plain and simple C implemented LLM inference, and/or x86_64 assembly implemented, and/or AMD GPU RDNA assembly.

Anybody?

It seems the author believes checking the return values of CUDA API calls is not "tiny" enough :-(

[flagged]

[dead]

[dead]