alt Hacker NewsWhilst processing in memory is clearly the future, I am unconvinced by this implementation.
Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2).
To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a 'around the chip' ring shift register to pass every element of one matrix past every element of the other.
Replies
Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.
People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts.
At this point processing-in-memory has taken on the aura of fusion power.
> Whilst processing in memory is clearly the future
How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.
Not if you have duplicates of rows on the first matrix, which can be done very efficiently if you build specialized hardware. Then its all just forward in parallel.
you're absolutely correct that pim without a real discussion about how that works in a broader communications context is kind of useless.
what I find strange is the adoption of a standard synchronous dram interface. that's a horrible left over piece of architecture that severely constrains the applicability of this device. control flow on the dram side can't initiate any transactions on its own, or respond after work has been done - its like usb, except with a hard limit on the response.
that severely limits the utility of the in-memory processors to doing things like encryption and compression - but even then those impose delays that effect the consistency model across that interface.
"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of support for the latest compressed/quantized number formats, which heavily limits the usefulness of being able to access memory directly. GPUs, even weak iGPUs, can dequantize/pad parameters on the fly which adds a lot of flexibility - and expose standard, well understood compute capabilities via CUDA, Metal or Vulkan. This is not quite comparable unfortunately.