alt Hacker NewsLlama-modchip: 284B DeepSeek on 64GB RAM, no GPU
Comments
Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855
To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
The quantization is probably the most important thing to know and its not clear at all from the dense word salad
At which quantization? That's the only thing I want to know.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
Stupid question, but does this AI-generated repo actually _do_ anything?
Comparing it on Nemotron, it runs just about the same as plain old llama.cpp on the cpu (except prefill seems much faster). I don't see an option to set up DeepSeek in "modchip setup", so I haven't tested that one yet.
Edit: this is on a Ryzen 7945hx 16-core Zen 4 + 96 gb DDR5.
I wish people would clean up the ai spammed emoji from the github readme. The readability is just horrible.
Interesting. There are quite a few forks of llama.cpp where streaming weights has been successful, or special quants / decompressing before processing.
- Bonsai Qwen 3.8 27B
- Strata
- Pulsar
- This...
I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.
I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.
So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.
I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.
At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.
Running a local LLM is similar in price to a mid-range photocopier.
It is about $10 per person per month for enough processing power for useful work with data sovereignty.
In a way this hardware shortage/price situation may come to be a helpful driver of efficiency and novel approaches. A lot of the 1980s demoscene guys came from countries where hardware was scarce and expensive, and it drove a lot of technical creativity.
How can CPU be fast enough? What does “modchip” refer to?
This is a very impressive achievement in an unexpected direction. I'm really looking forward to trying out qwen on this system.
This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.
I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded.
I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.
Very nice. I have been doing various SIMD hacks to my fork of llama, but some of these are ingenious combinations that I will be dropping into my ARM boards….
It works pretty good on my Ryzen 7 5700U/64GB DDR4
Steady 12 to 14 TPS.
Not blazing, by any means, but fast enough to get meaningful personal work done, with a power consumption of 15w.
Strata, Project Maya, the FP8 3x speedup on memory-bound workloads on Ampere cards and now llama-mod chip. I’m amazed with all these optimisations and thinking outside the box. Well done, don’t stop!
Anyone else finding these AI generated pages really hard to read?
Which DeepSeek V4.1 Flash were talking about? The 0731 that is benchmaxed to hell and runs worse than Nemotron for me, or the 1017 that needs triple the RAM that is really good?
Still, at 1.75bpw that is barely usable.
At least for me, the absolute minimum for the 1017 DeepSeek is 3.75bpw. Anything below and you're better with Qwen3.8-flash-next at 4 bits or preferably 4.5.
Tinfoil hat time:
I can imagine Open AI and Anthropic( among others) artificially manipulating the pc part market to keep consumers from buying capable hardware.
As is a ROG tablet is available for 3K with 128GB of ram. I don’t need to beat Opus, I just need good enough
Could I perhaps get an overview of how this works that doesn't require me to understand Claude-speak nor the jargon of domain experts?
Just because you CAN load it, doesn't mean you SHOULD load it. Sparse models n such ... but I digress.
hm... this is the first time. TIL something from the HN comments more than the actual article. I'm relatively new to the AI space and the idea of quantization is something I'll have to take forward when evaluating future models.
"2-bit quant" is the "in mice" of "look at how big a model I can fit on this hardware".
no intel AMX ? poor me
Prefill of this approach (prompt processing) is far too slow.
The readme is practically unreadable ai slop as well
all the pieces are coming into place for the Apple M7 chip, 2028 is going to be great for useful local inference
frontier labs are going to need to have some new tricks up their sleeve that are inaccessible by prosumer level compute
Do people remember humans read README.md files too or are we all just asking for the TL;DR; from AI slop by default?
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
The floor for a functional local AI system keeps getting lower, dispite ram and GPU prices continuing to go up. An E5 workstation with lots of threads, $300 of RAM and at least $100 of GPU gets you something that can do >100tok/s+ prompt processing, and ~20tok/s output with Qwen3.8-flash-next:IQ3_S. Toss a $500 GPU in there, and it goes from barely usable to pleasingly performant. For less than $800 you have something that holds its own with where the SOTA was less than a year ago.
I don't see how anyone justifies a claude subscription. Do you want to rent the tools you use to making a living from the people who are trying to automate away your job?