alt Hacker NewsWill the workloads exist? As in not going straight to something at least as dedicated as the Strix Halo CPU/GPU combo (with air quotes around the G)? Or the Apple max? Somehow I don't picture this as an attempt to make a full AI rig that just happens to be x86 in the housekeeping parts but just something that will make CPU inference a little less bad. In that case it would at best be a hedge against some low requirements use case becoming more important than expected, yet another unused spec sheet checkbox engineering marvel otherwise.
Replies
Dedicated PCIe or memory-attached inference units will continue to exist, and will continue to do the heavy lifting, but an ISA extension provides latencies any external unit would have trouble matching. You can, with some work, extract some decent throughput with CPUs alone, with a large enough CPU you can use for non-AI jobs the rest of the time. There was a nice writeup not that long ago here on HN describing the flags and the reasoning behind them to use on that specific machine.
Funny thing, I can't find the article.
Not long ago, China has demonstrated a supercomputer faster than any existing US supercomputer.
The Chinese supercomputer uses custom CPUs designed in China, which implement the SME matrix extension of Armv9.3-A, while the fastest US supercomputers use GPUs.
The Chinese CPUs with SME have a lower energy efficiency than the best NVIDIA datacenter GPUs, and also a little lower than the best AMD datacenter GPUs, but not only they reached a greater absolute performance, but they also reach a much higher percentage of their theoretical maximum throughput, and this is likely to happen for all problems solved on them, because writing efficient programs is easier for CPUs than it is for GPUs.
As discussed in TFA, the Arm SME and the Intel/AMD ACE are similar.
Thus server CPUs that have hundreds of cores and ISAs with matrix extensions can be very competitive in performance with GPUs, even if it is very likely that GPUs will always hold the records for the best energy efficiency.
The Chinese CPUs have fast HBM memory, with an 8 TB/s memory bandwidth per socket, equal to that of the latest AMD Instinct datacenter GPU and much greater than that of most less expensive GPUs. Very few consumer GPUs exceed a 1 TB/s memory bandwidth.
So at least where electric energy is cheap and abundant, running AI inference or training on server CPUs may be a competitive solution, especially when that computing cluster is also used for other purposes, not only for AI.