alt Hacker NewsThe relevant comparison isn't one mac studio to one RTX 6000, it's a 24 channel DDR5 system, which also has ~1.2TB/s of memory bandwidth (or more when Xeon 6 compatible 8800mt/s memory becomes widely available), vastly higher prefill due to more CPU horsepower, orders of magnitude faster networking, can hook into GPU accelerators, can be upgraded etc. A baseline 384GB system from eg Puget is ~30K vs ~12K for the 256GB Mac Studio and you do get value for the money.
Replies
So 3x more, plus the cost of a GPU (another 10k?). How is that value for money to get slightly better performance?
Excellent post. Heck yes. And with MRDIMMs coming, we're going to get another >50% boost in throughput per channel real soon, with a massive uptick in max capacity (4x).
It feels like PCIe is a bit of a boat anchor here. There's a SATA->NVMe style transition waiting in the wings to make this all so much better. We really need post-PCIe GPUs. CXL with it's very small low latency flits. This is an "almost certainly not" but I wonder if you could mix PCIe and CXL so you could have the GPU memory expose vmeme as a bunch of CXL.mem pools but still have an otherwise pretty normal GPU. It seems madness that UALink went all in on GPU-to-GPU with no affordances for connecting to host computers.
Prefill is gated by GPU compute, not CPU compute. A Epyc/Xeon CPU will never beat an M5 GPU in prefill. If you want something that can match M5 Ultra in prefill, you'd need to add beefy Nvidia GPU. However, the problem is that the GPU memory is separate from the 384GB system memory. And here lies that advantage of Apple Silicon. It's unified memory and accessible by the CPU and GPU.