alt Hacker NewsI was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.
To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)
In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.
Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.
Replies
Not sure if mmapping is the right way. In my own tests I noticed that simple mmapping will produce many small reads and not keep the SSD queue saturated. So if RAM is large enough to cache most experts, that is a rounding error. But if the base weigths without experts fill more than half or RAM and you basically need to load in a few experts for each layer of each token, the latency gets important and mmap sadly blocks until the data is loaded. You can't do concurrent requests for multiple experts with mmap (but you know all the ones you need right after the router ran). And even going one step further, depending on the arch / the tensors you need, you could eagerly load some, start computing with them and load the rest of the expert tensors in the background (extra thread or async io) parallel to the compute. This is not really possible with mmap, even with madvise.
One further step is predicting which experts will be needed next token / next layer. LRU does this okish. But a learned projection from the hidden state can do better. Or even a simple correlation from past activated experts. Expert usage is heavily skewed.
if you like, colibrì always needs to improve so if you have ideas or anything else you are welcome for pull request issues and also benchmarks!
This is the approach I was wondering about.
> Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.
Posting on the Hacker News forum is not worth risking your, or anybody else's, life.