Not long ago, China has demonstrated a supercomputer faster than any existing US supercomputer.

The Chinese supercomputer uses custom CPUs designed in China, which implement the SME matrix extension of Armv9.3-A, while the fastest US supercomputers use GPUs.

The Chinese CPUs with SME have a lower energy efficiency than the best NVIDIA datacenter GPUs, and also a little lower than the best AMD datacenter GPUs, but not only they reached a greater absolute performance, but they also reach a much higher percentage of their theoretical maximum throughput, and this is likely to happen for all problems solved on them, because writing efficient programs is easier for CPUs than it is for GPUs.

As discussed in TFA, the Arm SME and the Intel/AMD ACE are similar.

Thus server CPUs that have hundreds of cores and ISAs with matrix extensions can be very competitive in performance with GPUs, even if it is very likely that GPUs will always hold the records for the best energy efficiency.

The Chinese CPUs have fast HBM memory, with an 8 TB/s memory bandwidth per socket, equal to that of the latest AMD Instinct datacenter GPU and much greater than that of most less expensive GPUs. Very few consumer GPUs exceed a 1 TB/s memory bandwidth.

So at least where electric energy is cheap and abundant, running AI inference or training on server CPUs may be a competitive solution, especially when that computing cluster is also used for other purposes, not only for AI.