alt Hacker NewsWhich is more important for AI/LLM stuff, the number of GPU cores or the amount of unified memory? I can choose from a Mini with the M5 Pro 18CPU/30GPU and 64GB RAM or a Studio with a M5 Max 18CPU/40GPU and 48GB RAM for roughly the same price
Replies
Memory bandwidth is one of the most important factors. The Max machines typically have higher bandwidth than Pro (you should look up these exact models to verify though). Having said that, you could run bigger models with 64GB.
I went with a 64GB M2 Max a few years back even though it stretched my budget and I'm glad I did. I'm not sure what I'd pick in this case. Tough call.
Larger model -> more memory
It depends on what model(s) you want to run.
They affect different things. You want more powerful models -> more memory. You want the models that run to be faster -> more gpu cores.
They are both somewhat important. The compute speed impacts the tokens per second, and memory impacts the size of the model/capability.
IMO memory size is the most important since you can always wait longer for it to compute while a blazing fast but dumb model is much less useful.