alt Hacker NewsYet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
Replies
infecto • today at 1:36 AM
Go back and correct your idea that consumer hardware made asics obsolete. Then we can figure out if asic or asic like devices for inference will have no advantage.
andy_ppp • today at 1:33 AM
Except Taalas is much faster than GPUs, orders of magnitude so. They aren’t going to get 100x faster at inference any time soon!
mitxela • today at 1:49 AM
Yeah! Nobody needs chatjimmy.ai. Nobody needs their results to come back instantly instead of at 10 tokens per second. Nobody needs a CPU faster than a megahertz.
ASIC vs GPU doesn't make a ton of difference when both are relying on commodity DRAM; in that sense, LLMs are more like the anti-ASIC cryptocurrencies. But the actually interesting ASICs are the ones that ditch the commodity discrete DRAM chips. They lose out on the memory density and thus struggle to scale up to the largest models, but for what does fit onto a Cerebras wafer or a Taalas chip, the speed is phenomenal. They have a real shot at securing the "smart enough, and really fast" segment of the market.
And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.