logoalt Hacker News

simpetre • yesterday at 6:18 PM • 1 reply • view on HN

Modern NVIDIA GPUs contain tensor cores that are optimised for MMA operations as well though right? The standard CUDA execution path isn’t comparable, but the tensor core path very much is?


Replies

Melatonic • yesterday at 7:46 PM

I'm no expert but I think we're dealing with two different ends of the spectrum here. Nvidia GPUs are likely cutting edge within the individual board / processor itself. Their main limiting factor right now seems in effectively networking them together (which is why they bought Mellanox or whatever awhile back to further develop Infiniband).

TPUs are significantly less flexible and I would guess at an individual level not as efficient or fast as Nvidia. But they're designed specifically to be bundled together in large arrays with similarly large data pipes to storage and interconnects to cache, ram and other compute. I would guess Google didn't bother trying to sell them until recently because you needed to be deploying them at large scale for it to be even worth also developing the software for.

What we really need is innovation in super fast (bandwidth and latency) and reliable interconnects. I suspect the days of slotted in GPU and RAM will be over soon and we'll see a move everywhere to pin based sockets (like we've always had for CPUs). Stuff like SXM and LPCAMM.

We've started to vertically stack ram out of necessity but have yet to really move on from the 2D plane for the motherboard itself. Slots allow more devices to connect to one flat board. But if the board itself had more surface area (by being maybe an L shape or box) you could slot in a lot more chips physically