I think OP is overstating it a bit tbh. So Nvidia has the market for the hardware, which helps, but there is literally no alternative to CUDA. Nvidia keeps it a scalpel for skilled users, its not super easy to use, but unlocks orders of more magnitudes of power for the use cases it excels at vs CPUs. I don't have access to anything like it in the Apple ecosystem.

AMD has had years to try and counter it, but just has not. Google is kinda trying to do an end run around it with TPUs but they are still niche high end stuff with limited availability.

Its really just CUDA, and CUDA can be seen as somewhat akin to C for assembly used by Nvidia's gpus- In many ways a wrapper around the low level hardware that often has those details bleed through.

To paraphrase the apocryphal Winston Churchill quote about democracy:

“CUDA is the worst development ecosystem in existence. Except for all the others.”

Retraining a large high paid user base is often a non-starter. To put this in perspective, Boeing’s eventual retraining costs for all the pilots for the 737Max was around 5 billion dollars.

Looking at software more specifically the Linux foundation reported based on software dev salaries in 2008 it would be 1.4 billion to only write the Linux kernel.

Up until about 2023 there wasn’t enough money involved to have any reason to make a real CUDA killer even if you could get it adopted.

One of the rarely-mentioned value adds Nvidia provides is nccl[1] which makes multi-node networking and topology essentially plug and play. The other players have since caught on [2][3] and are working hard to catch up but I'd say networking is a real moat.

[1] https://developer.nvidia.com/nccl [2] https://pytorch.org/blog/torchcomms/ [3] https://rocm.docs.amd.com/projects/rccl/en/latest/

It's a totally reasonable question, and one that everyone asks when they're learning about CUDA. The frustrating answer to your last question is that lots of companies have shipped GPU dev environments that can theoretically be used instead of CUDA. AMD has ROCm, Apple has had a couple projects (OpenCL, Metal), Intel has some stuff, and there are newer efforts like TinyGrad + a generation of slightly higher level frameworks from AI companies, like Triton from OpenAI.

The basic problem is that CUDA has become something of a Schelling point. If you want to train a model right now, the highest performance you can get is almost certainly on CUDA. From the basic general matrix multiply operation, to specific NN architectures, CUDA is going to have incredibly optimized implementations out of the box. And it's going to make multi-GPU training so much easier. And all the dependencies you build on (those layers you import from PyTorch or Transformers or whatever) are going to work optimally right away on CUDA. And that weird random repo that you found with a unique optimizer--it runs on CUDA too. And now the cool new implementation that you're about to release is also going to be built for CUDA.

It's so tempting to think "Just write replacement software", but you also need to transition the entire ecosystem in large part to match CUDA's effectiveness, and you need to get comparable performance out of your chip/library combo as NVIDIA can get out of its cards with CUDA.

There's a whole story here to how effective NVIDIA has been at navigating this. Very early on, they heavily prioritized PyTorch and TensorFlow, getting involved in the projects as much as they could and making sure they always ran best on CUDA. But the TLDR is that yes, you're right, another company could write a CUDA competitor. But actually replacing CUDA is a much larger task.

I'm personally hopeful that with the rise of coding agents, we see more movement on this front with other projects moving into view. It will take some time for any ecosystem to start to emerge that can dislodge CUDA for researchers who don't want to dive that deep into the stack, but hopefully we start to see some momentum build.

CUDA has many problems, but I would say less problems than ROCm, etc; even before considering the ecosystem and that more people (or open source projects) have already solved CUDA's problems for you.

Ironically, there was an open source project that was making great progress on CUDA compatibility on AMD hardware. AMD hired the lead developer, and then he shut down the project.

Software has always been the moat but for some reason it's always hamstrung by upper management. The latest of the frenzies being replacing sane (or whatever we have) of development practices with AI-slop.

Management likes it because it removes software developers from the loop.

[dead]