alt Hacker NewsNvidia's biggest advantage in AI has never been only their hardware performance but how entrenched their software is in ML research that flowed down stream. However, if you've actually used CUDA C/C++, it's pretty one of the worst software development ecosystem imaginable: you get all the footgun of regular C++, plus GPU compute pretending to be C++ and but doesn't actually behave like C++ because CPU and GPU compute are fundamentally different, and the only reason people put up with it is because Vulkan and HIP C/C++ are even worse.
Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board that people can plug in to their PC for local development and sane low level API to develop against, instead you have to go through their cloud and their full software stack which greatly limits ecosystem growth. The minute that Google figures that out, that's when Nvidia's dominance would be challenged.
Replies
That's really interesting. I have no experience writing anything that involves GPUs/TPUs, but over the years I've consistently read that CUDA is the "real moat" of Nvidia, which I never totally believed, but the way you describe makes it seem like it's not actually a moat in the slightest. It just happens to be an ecosystem associated with hardware that is not only considered the gold standard but happens to be more open than potential competition. Could it be that Nvidia has been on top because none of the competition has actually tried kicking them where it hurts?
The biggest advantage of tpus is the high bandwidth fiber optic interconnect between them that allows distributed computing on pods with thousands of tpus and the co-design of cooling systems that go with their racks. I do not think that we will see personal tpus any time soon.
> Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board that people can plug in to their PC for local development
I’m not familiar with this field, but to my brain, https://www.amazon.com/s?k=Google+Coral seem to show me several such options.
Now with LLMs why a programming framework is a moat?
I had a hard time understanding why didn’t AMD make a better developer experience for this two years ago, and am now even more baffled that even with all the LLMs they still don’t seem to have moved a single inch, despite this probably being a tens of billions dollars worth feature.
I’m not entirely sure if local development will lead to Nvidia’s supremacy being challenged.
I think a simple reason why it’s been hard to unseat in Nvidia is first mover advantage. A lot more water has flown through Nvidia pipes than TPUs or AMDs chips for that matter.
TPUs and AMD chips aren’t priced cheaper than NVIDIA (at least for my purposes training models). So there hasn’t been an impetus for me to venture there and use those chips.
Anecdotally, folks I know who have tried using TPUs and AMD chips have hit more issues with the underlying drivers than with NVIDIA chips. That costs time and money to fix.
Eventually the other chips will go through enough iterations and stability will be reached
Are the switching costs of CUDA ecosystem potentially threatened because LLMs are now quite good at transcoding into other languages? In other words, is Nvidia's greatest strength (AI) also potentially its undoing?
> Google's limitation is that they still don't offer TPUs in a PCI-E card/dev board
Nvidia's sells hardware yet their market cap is about the same as Google's.
How much value could Google get by selling hardware too? Google'd be selling to competitors, so difficult to capture much of the value and would decrease Google's value as an AI company. Maybe a child company?
Genuine question. Given that LLMs are supposed to allow us to rewrite anything, and I am an LLM believer, what I don't understand is: how does CUDA continue to be a moat in a world where LLMs can rewrite entire software development stacks? If NVIDIA is right about AI, isn't this same technology going to erode the software side of this same software moat?
Interesting take on Google's TPUs. What I've previously heard (and still believe) is that Google's decision to only rent out, never sell, their TPUs is a deliberate and savvy strategy for bolstering GCP, which will work provided that TPUs are able to actually compete with other hardware (in practice meaning Nvidia). A few months ago there was some discussion on HN comparing them, and I think the verdict at the time was that their latest-gen TPUs win on compute-per-Joule for LLM-type workloads by quite a margin, which I think is huge for those who want to run LLMs at scale.
I mean they had/have the Coral but that's in an entirely different market segment
The CUDA runtime coming with a gazillion reasonably decent kernels (DNN, BLAS, CUTLASS) and a concurrency system (NCCL) is a big deal; especially in the “early days” very few researchers or development runtimes were even writing their own kernels or dealing with CUDA C++ extensively, they were wrapping the ones NVidia gave them.
I do agree that it’s really not great, and I also have never been a strong believer in the CUDA moat overall; as the need for GPUs moves from research to production (inference), companies are plenty willing to build software from scratch anyway (and we see this with AMD GPUs being in plenty high demand in the datacenter and enthusiast market now).