alt Hacker NewsI'm hoping to see progress in this space.
Folks talking about how 32G is not enough for local use, but then there's been work like this to empower it.
My hope is that the new 32G M6 will be "useful" locally, possibly because of work like this.
Replies
trollbridge • yesterday at 9:16 PM
32GB is simply too tight; you need 8 minimum for the OS and you need about 4-8 more for the LLM you’re visiting and kv cache.
carloslfu • yesterday at 5:51 PM
yes! I'm bullish on this. there is a lot of work to do. I've been experimenting with pruning, distillation, and retraining too. I'm sure your 32gb m6 will run a badass local model!
Yes, but also 12 tok/s versus Claude is so far from comparable. I know that it’s not exactly 1:1, but it’s a long way from an easy trade-off, especially considering hardware prices for high levels of RAM.