logoalt Hacker News

notnullorvoid • yesterday at 5:00 PM • 10 replies • view on HN

It's already happened hasn't it. Google's TPUs are apparently much more efficient at inference workloads. However the hardware isn't available to the public like Nvidia chips are.

I'm unsure about the energy efficiency of Tenstorrent cards, but they are at the moment much more cost effective than Nvidia. But software support is far behind, I suspect if it wasn't the cost would be more equal.


Replies

14113 • yesterday at 10:00 PM

Modern NVIDIA cards are essentially TPUs - especially those spec'd out for AI data centres. TPUs aren't anything particularly special - they are just large vector processors, with hardware specialised for the data + instruction fetch patterns that are seen when running AI models. The graphics cards that NVIDIA sells for data centres are just large vector processors with hardware specialised for the data + instruction fetch patterns that are seen when running AI models.

Melatonic • yesterday at 7:36 PM

Google has two different types of TPUs now. One for inference and one for training.

Do we actually know they're more efficient per watt or for the power used and heat produced ? Or are they just easier to scale (and manufacturer) en masse ?

There's also the CUDA moat which is hard to get around

➕ show 3 replies
epistasis • yesterday at 5:48 PM

Inference has two steps, prefill and decode, with very different requirements on the compute to memory-bandwidth ratio.

Already these are run on different hardware, at least as I understand, and the computational needs could further drive differentiation.

woodrowbarlow • yesterday at 5:59 PM

there's also stuff like Talaas, where the ASIC is the actual model weights

➕ show 2 replies
dominotw • yesterday at 5:08 PM

Doesnt google themselves use nvidia chips. why do they need nvdia chips then?

➕ show 2 replies
flowerlad • yesterday at 6:21 PM

> However the hardware isn't available to the public

It is available to Oracle.

https://www.oracle.com/news/announcement/ai-world-oracle-bri...

doctorpangloss • yesterday at 8:22 PM

This may be, but look at Alphabet's share price:

jgalt212 • yesterday at 7:32 PM

> However the hardware isn't available to the public like Nvidia chips are.

Why? Don't they want to add another $3-5T of market cap?

hrncn • yesterday at 5:24 PM

You have basically managed to sum up why NVDA keeps dominating, no on can "just buy a few TPUs" and that has been the case since the first TPUs were announced years ago (yes yes GPU shortages and so on, but no one questions general availability of GPUs), and the current Tenstorrent model support page looks like an invitation to beta test HW that might (or might not) be able to run current and future models at some point.

But the next gen of Tenstorrent HW looks great on paper - problem is, I could have written that before the current gen was released.

All of the allegedly competing HW looks like that and has looked like that for years, either a lack of availability, capability, or both.

trollbridge • yesterday at 5:12 PM

TPUs aren’t as effective at training.

Inference has already been commoditised.

➕ show 1 reply