logoalt Hacker News

Melatonic • yesterday at 7:36 PM • 3 replies • view on HN

Google has two different types of TPUs now. One for inference and one for training.

Do we actually know they're more efficient per watt or for the power used and heat produced ? Or are they just easier to scale (and manufacturer) en masse ?

There's also the CUDA moat which is hard to get around


Replies

jordanb • yesterday at 9:26 PM

Cuda is a moat for traditional data science, where you have a torch model and you want to easily run it on a GPU. CUDA is the happy path for doing this.

Nvidia's moat for LLMs isn't Cuda, it's installed capacity, arrangements with frontier labs, TSMC bandwith, and just the fact that it currently has no real competition.

➕ show 1 reply
SwellJoe • yesterday at 9:03 PM

"There's also the CUDA moat which is hard to get around"

Is it really, though? Porting to a new architecture is an ideal use case for LLMs.

➕ show 1 reply
cyberax • yesterday at 10:45 PM

CUDA is useful if you want a human expert writing new experimental kernels, perhaps while testing different algorithms.

But once you do that, translating CUDA into one of many available vectorized backends is trivial.

We have a bunch of custom kernels for geometric operations (like merging polygons) and they were rather difficult to write a while ago. AI rewrote them in HIP during my lunch break.