alt Hacker NewsDedicated PCIe or memory-attached inference units will continue to exist, and will continue to do the heavy lifting, but an ISA extension provides latencies any external unit would have trouble matching. You can, with some work, extract some decent throughput with CPUs alone, with a large enough CPU you can use for non-AI jobs the rest of the time. There was a nice writeup not that long ago here on HN describing the flags and the reasoning behind them to use on that specific machine.
Funny thing, I can't find the article.
Yeah, I guess in these days of hardware not getting completely outdated even after half a decade, it might well make sense to spend chip area on specialist units that are shut down almost the entire time.
In the old days, when it was all about squeezing maximum performance from a given chip area budget you'd prefer using those transistors for some improvement of the general purpose parts, no matter how marginal. But now it's the thermal envelope, or power consumption of you prefer looking at it from that angle, and even slightly esoteric special purpose units can start making sense.