alt Hacker NewsThe tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applications noted above eventually got their own flavors.
I think the Von Neumann bottleneck is mostly a feature. The fact that communication of information across distances is expensive should not be immediately assumed to mean that it is universally flawed to do this. You are paying for something when you use all those joules. I'd argue we are usually wasting our energy with regard to information communication (e.g., lighting up a network interface & copper because we couldn't be bothered to use SQLite), but other times this stuff is fundamentally required for practical solutions to exist.
Replies
Yes, and no.
Two decades ago it was a challenge to get people to see that what they were doing was preventing horizontal scaling. Today horizontal scaling is table stakes and people don't even always register that they are doing it. It's just how we do things, no thoughts.
PIM requires problems to be decomposed into horizontal scaling problems. Then what you should do with PIM is take a problem that used to be solved by 2 racks of computers and squeeze it down to less than half a rack by stuffing a bunch of these into a single box to do 8-10x as much work per box (and double the cluster size to offset Jevons' Paradox because it's so cheap now that you'll do 2x as much of it)
In a world where AI is writing all of the code, the difficulty of the task may no longer be a blocker.
There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.
I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module.
Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region.
Let's say you would need to run a sum over megabytes of data like for accounting. You could just ask RAM to do it and load just the result. The bandwidth would could be 8x higher and software could stay the same.
Doing scalar operations, frequent dereferencing and similar would not get much peformance benefit in many cases, as loading and accessing CPU cache is often much faster. But simple vector operations over large data could be massive.
Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.
Personally I'm a big fan of the "in-ram accelerator" idea especially for server space. Doing fuzzy search in RAM could be massive performance improvement.