> Gemma 4 31b? Firstly you don't need 64GB for that model.
You don't? It for sure doesn't run on my 32 GB M2 MAX.
What quant? You should have no problem running it at Q4 with 256K context, Q5 or Q6 even although maybe not at full context. I can run Q4 on a 4090 with just 24GB VRAM.
alt Hacker News
What quant? You should have no problem running it at Q4 with 256K context, Q5 or Q6 even although maybe not at full context. I can run Q4 on a 4090 with just 24GB VRAM.