Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't

> Put two machines on a desk, each about $2,000. One is a tower with an NVIDIA RTX 5090: 32GB of the fastest consumer memory ever shipped, 1,792 GB/s. The other is a mini PC the size of a paperback, an AMD Ryzen AI Max+ 395 "Strix Halo" box with 128GB of soldered memory at roughly 256 GB/s.

Doesn't change the conclusions of the article, but each of those machines is more like $4k+

https://www.microcenter.com/product/711961/amd-ryzen-ai-halo...

I'm writing my own inference engine for Strix Halo and the same model. I already have 30%+ performance plus a more graceful decay over long contexts; that said, their point stands: memory bandwidth is what you really want.

"integrated graphics processor, using system memory" had its name dragged through the mud for decades.

So we had to rebadge it to "unified memory".

Curious if we'll ever see some old integrated graphics processor "hacked" to manage to handle 128 GB of allocated system RAM and be able to serve diffusion-LLMs at a decent rate on "old" hardware...

For some reason, this reminds me of my last shared memory system. It was an Athlon XP 1800+ with VIA ProSavage back around 2002. It was just barely able to run CS 1.6.

Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically

Why would a RTX 5090 with 32 GB not be able to deal with a 40 GB model? Is there anything preventing me from swapping the weights that do not fit into VRAM in and out of RAM? PCIe 5.0 x16 should max out around 64 GB/s, so slower than the unified memory machine, but at least it should be possible.

I'm going to say this that we're not even close to the limits of what actually needs to be accomplished so at some point, memory will start needing better tiering for inference some day ....

If compute is not the bottleneck, memory is easy-ish to produce (the hard part is mostly on the fab side); what stops a Chinese NVIDIA (huawei) from being 10x cheaper?

"Can't" is not really correct.

Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.

Do unified memory CPUs suffer from the same memory shortages as normal memory?

I guess they're just welding the memory to the CPU chip, but still curious.

The current “big GPU” has 96gb of memory, but that’s not a “consumer GPU” apparently, while a $5000 Spark is a “consumer PC” I guess. In any case you’re probably better off running a large open weights model on the cloud.

Uhh the 5090 alone is double the cost of their quoted PC prices.

Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).

Let's also ensure the SSD doesn't age prematurely.

[flagged]