alt Hacker NewsMy 5090 has a bandwidth of 1,792 GB/s, and that is just about good enough for basic (i.e. 27B models) local LLM inference.
The fastest M6 MacBook Pro (24/32GB configs) has a memory bandwidth of 170 GB/s. And the M5 Max is 614 GB/s.
Am I missing some nuance of the M6 MBP here, or am I right to assume it won't set the local LLM world on fire?
Replies
Isn't the laptop version of the 5090 is a more apples-to-apples comparison? It has 896 GB/s memory bandwidth (compared to the 614 on the M5 Max MBP).
The 5090 does have a bandwidth advantage, but also must pass more data from main to GPU memory for any particular task. Comparisons of a 5090 to a DGX Spark at 273 GB/s are illustrative since the Spark uses unified memory similar to Apple's M-series. [0] The 5090 certainly wins most benchmarks but that doesn't mean the DGX Spark is not useful.
0. https://float16.cloud/en-en/ai-benchmark/dgx-spark-vs-rtx-50...
The case of your 5090 having become a coveted artifact of great rarity in these times - may be a nuance of some measure.
Also, the following two have very different 'needs' to feel good enough:
1. heavy agentic workloads via harness vs 2. a simple private chat interface with a local instance
Even if an MBP's bandwidth were higher, it's tough to see a reason to run it as a constantly loaded 24/7 inference machine due to heat degradation.
P.S. still extremely awesome to be able to host and chat with a decently quantized 27B model on your laptop. ...and to be fair even a 32GB Air would do that! It is a thing of beauty: the amount of folks this may add to the hacker/tinker community.