alt Hacker NewsProbably! But not viable yet; the chips would be about a year behind SOTA. Note the ~16 months that the article quotes as being insanely fast to get this chip to tape-out (read: start producing). We'll have to bootstrap our way there: AI is actively being used to get us closer to viable lead times for this.
Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a month, start to finish, for the physical processing.
Maybe once LLM improvements asymptote further?
Replies
How much of that 16mo is design versus just production? If there was a “plug and play” chip where you just BYO weights, how long would it take?
The bigger issue seems to be that these chips can’t hold that many weights at the moment.
(I’m curious if chips with large weights in them would be more tolerant or less to yield issues. If you flip a few bits in the weights, does it really matter at scale?)
They could etch the model architecture, without the weights into the chip.
This way newly post-trained model can be loaded and served the same day.
> Maybe once LLM improvements asymptote further?
Maybe! But it also doesn't require the rate of improvement to slow down. As long as some current model is eventually "good enough" for general use, it could still be a market-killer at a very low marginal price thanks to ASIC. Even if slower, much more expensive models are 10x better, that doesn't actually diminish the utility of the ASIC model, as long as it's "good enough".
tapeout could shrink but days per mask layer (DPML) does not have much margin..
It may not matter. Think about why SOTA model companies are exploring chips. What do chips offer?
If SOTA models haven’t peaked, then the SOTA model companies would still be churning out better and better intelligence.
The metal masked ROM is basically only 2 metal/contact layers. It's not a full new design and tapeout. You could roll a new set of parameters every ~2-3months. It's not an architectural change. See statements below.
https://www.eetimes.com/taalas-specializes-to-extremes-for-e...
https://www.turingpost.com/p/taalas
https://cambrian-ai.com/taalas-launches-hardcore-chip-with-i...
Part of the key is that by moving even from 6nm to 3-4nm one could embed a 20-30B model as part of a MoE (or only a subset of activated layers) on a single reticle die (note B300s are already multi-reticle), with a separate predictive/dispatch model controlling them each on a separate chip. This is without even stacking CiM ROM die. Moving the layer activations (and KV cache etc) between die requires relatively high speeds (and low latency), but distributed with multiple die in parallel might well be doable even with standard multilane PCIe. Of course KV cache prefill could also be handled by external GPUs. I'm sure AMD will make some reasonable choices.