Practically ~20GB with KV cache

> We quantize weights to ~4-bit, bringing the LM under 20 GB. We validated minimal to no degradation on agentic tasks under compression.

https://www.reddit.com/r/LocalLLaMA/comments/1vkgsum/introdu...

I think if there's going to be advantages to making smaller, more targeted models, those advantages will probably come from targeting specific domains, not from targeting specific languages.

I think that if an LLM can't abstract over the differences between Python and C++, it probably will have an even harder time abstracting over the differences between writing code that manages a webserver, and writing code that does aerodynamic simulations.

I am running it on a single RTX 3090 (24GB VRAM).

Some folks on Reddit are having the same experience: https://www.reddit.com/r/LocalLLaMA/comments/1vkm42m/muse_gl...

It uses an order of magnitude less VRAM at longer contexts which is a huge advantage over Qwen 3.6 27B

Coming from the PC games industry in the 90s and early 2000s, it was a struggle to run some of the games on release. 90%* of people wouldn't be able to play the AAA games on release (think Crysis, etc). This period of local LLMs reminds me of that time, whereby the hardware just isn't there yet. Give it time, and the prices will drop.

* total guess

I'm sooo happy I pulled the trigger on upgrading and getting a new laptop (with 64 GB RAM) last summer. Feels like it was just in time before the exponential price jumps.

There have been discussions on language specific not really being a relevant change to reduce size.

Well if you're spending thousands on API tokens already, you could just drop the same amount on a 128GB MacBook Pro and that's a one time cost.

I feel like we’ve had this discussion before. From what I remember, specialized models rarely do that much better than general ones, hence no mode Codex models.

There is no good reason to believe language-specific models are going to be any meaningfully smaller, just worse. Same as English-only models vs those trained on a multilingual corpus.

> 64GB Macbook pro with an M5 chip costs more than 4k Euros in Germany

Sure, if you want the latest and almost* greatest. You can pick up an M1 Max 64GB for ~1k.

* I guess 128GB also exists

The machines that can run this are pricey, but not beyond a high end developer machine.

4K bucks buys you around 180 months of <insert AI subscription here> with zero upfront cost.

Even if you had a 64GB machine: Are you willing to reserve 90% of your memory to run a LLM? With dirt cheap models like deepseek-v4-flash that will run "forever" on $10, the answer for me is clearly: no.

I don't understand the desire to run own AI models for programming locally. No laptop is ever going to be as powerful and energy efficient to run anything close to OpenAI, Anthropic or Google models. A model you can run on a loptop is simply not going to work as well as it's needed for programming. Small models for linguistic work fine, but anything more sophisticated simply won't provide enough resources or power. Or models would need to be significantly dumbed down - then why use them at all? So far the idea of carrying a "thin" or "thin"-like device looks more reasonable to me, while running AI on your own server.