alt Hacker NewsOff-topic and somewhat of a rant, but I'd far prefer us all focusing on open standards like HIP, SYCL, OpenCL, etc.
It's unbearable that most LLM inference happens on closed H/W, closed drivers, and closed SDKs.
Replies
The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.
yeh - exactly.
Sucks like important libraries like Alphafold are locked into CUDA. Its ridiculous for researchers.
Commercially it's better to have AMD support CUDA which can help break Nvidia soft monopoly.
I would too, but sadly that's Khronos' job to organize, and they've had trouble getting American vendors to work together.
It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.
it will only get worse now when hughingface was bough by nvidia, not immediately, but in span of year or years...
It's impossible to have an open standard. Hardware accelerators are nothing alike and have different perf characteristics. Each kernel is tuned to the hardware. The idea of writing a performant kernel in opencl is a fantasy.
Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.
On the SDK front, you tried just having an agent reimplement the model you're interested in and just use the weights? I've taken to treating off the shelf implementations as reference implementations anyway, because I can often squeeze out significantly better performance for my configuration and use case by having Codex hammer at it for a few hours.