alt Hacker NewsMy main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
Replies
I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
Now many mini-PCs and desktops are able to read simultaneously from 1 PCIe 5.0 SSD and 1 PCIe 4.0 SSD. This can ensure a reading throughput around 20 GB/s, i.e. 20 times faster than on author's system.
With only 1 PCIe 5.0 SSD, the reading throughput is still significantly more than 10 times faster than on author's system.
So it is likely that inference speeds around 1 token/s are achievable on something like a NUC mini-PC.
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.
In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.
> on hardware that ordinary people can afford
These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
sigh
[flagged]
The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028.
(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)