Google has two different types of TPUs now. One for inference and one for training.
Do we actually know they're more efficient per watt or for the power used and heat produced ? Or are they just easier to scale (and manufacturer) en masse ?
There's also the CUDA moat which is hard to get around
"There's also the CUDA moat which is hard to get around"
Is it really, though? Porting to a new architecture is an ideal use case for LLMs.
CUDA is useful if you want a human expert writing new experimental kernels, perhaps while testing different algorithms.
But once you do that, translating CUDA into one of many available vectorized backends is trivial.
We have a bunch of custom kernels for geometric operations (like merging polygons) and they were rather difficult to write a while ago. AI rewrote them in HIP during my lunch break.
Cuda is a moat for traditional data science, where you have a torch model and you want to easily run it on a GPU. CUDA is the happy path for doing this.
Nvidia's moat for LLMs isn't Cuda, it's installed capacity, arrangements with frontier labs, TSMC bandwith, and just the fact that it currently has no real competition.