alt Hacker NewsThe open weight, open architecture releases of the past several days has me more convinced that ultimately, the winner will be whoever burns their models to ASICs fastest.
The LLMs themselves are capable of doing some aspects of chip design as evinced by the K3 press release.
Furthermore, the frontier models are "good enough" for a wide swathe of tasks and will soon hit that threshold for a good amount of software engineering (if not already). Does anyone think we need a Mythos level model to plan a road trip, or give someone tips on making a cake recipe?
A Fable 5 model running at 9,000 tokens/s on an ASIC rather than 150 tokens/s on electricity chugging Nvidia GPUs, or even giant SRAM Cerebras or Groq chips could be good enough to meet the majority of demand.
Furthermore, if you're an enterprise the risk of data exfiltration and feeding data to a potential competitor like OpenAI or Anthropic is greatly reduced if you could shift to on-prem ASIC deployments. A handful of chips could cover a wide variety of use cases and cover them more securely. There are a lot of corporate use-cases for LLMs that are not frontier math research or coding.
Replies
A Fable 5 model running at 9,000 tokens/s on an ASIC rather than 150 tokens/s on electricity chugging Nvidia GPUs, or even giant SRAM Cerebras or Groq chips could be good enough to meet the majority of demand.
> Does anyone think we need a Mythos level model to plan a road trip
Yes. The latest OpenAI and Anthropic models are terrible at planning roadtrips.
This is something I try to use them for frequently. They constantly get things completely wrong.
I’d say that about half of the stops they suggest fail to follow whatever filters I’ve asked for.
> Does anyone think we need a Mythos level model to plan a road trip, or give someone tips on making a cake recipe?
This is starting to look at a lot like Intel vs Arm from the last era.
The Fable & Mythos are starting to look like a giant Xeon, while the smaller lighter models are starting to look like a lot of tiny ARM chips which sip on power instead.
The risk is the same as what Intel had. There is a group who are pushing them to go bigger and with a resource no limit approach, who have a lot of dollars to push you that way.
Follow them and they lead you to a pile of money, but then you risk something like Apple Silicon happening.
Something which got better because of efficiency & continuous improvement, not neutered due to it.
> whoever burns their models to ASICs fastest.
There is already custom hardware see cerebras.
GPUs have a lot of slack there is at least one lab that had a (small 8b) model generate almost 3000 tokens per second on a MI300X for a talk, instead of the typical software stack that did maybe 100ish tokens per second.
High bandwidth flash storage is in the works, i.e hard drives with TBs of storage and over 1 TB per second of read speeds. Meaning that in a couple of years you may be able to buy a card with 40-90GBs of HBM and 4TB of HBF and run a 3T model locally at a reasonable speed for 10-20k as opposed to a cool mil.
> the winner will be whoever burns their models to ASICs fastest.
It'll obviously be China, and they won't need the bleeding edge of lithography tech to make it happen. Every Chinese smartphone will have something like Sonnet 5, along your car (well, not those of us in the US, but we'll look longingly at pictures of them while we drive whatever the government decides we're allowed to drive in Fortress America).
Give it ten years and your smart litterbox from Temu will be running its own local model.
While running the model at 9000 tokens/s is the more flashy demo, I imagine running 1000 concurrent requests at 150 tokens/s each is the much more achievable goal
I'm convinced there is a Transmeta/transpiler solution somewhere between 9000 t/s ASIC and Nvidia GPUs. LLMs fit the model much better than what they were trying to do with general purpose in the 2000s.
My gut is that, at least for now, there's a timescale mis-match problem. Hardware still takes too long, then you have to deploy it. I don't know much about "burning asics", but if the whole process of spinning up programmable GPU data centers is months, I imagine the whole ASIC cycle has some catching up to do. To be clear, by timescale mismatch I mean model quality improvement timescale vs. deployment timescale. But maybe we're finally getting to the point where behind-the-frontier-but-cheaper are in sufficient demand and ASICs make it cheap enough to close that gap.
The field is moving too fast for it to make sense. ASICs have a 12-24 month development cycle and you’d have to throw them in the trash 4 months later.
We're getting new models every three months. They'd have to work quickly to get those chips made and installed.
> the winner will be whoever burns their models to ASICs fastest
As of today, that appears to be Google!
Yeah, I have been thinking exactly the same. Makes all the sense in the world and is inevitable as far as I can see.
They require a lot of computation regardless of whether this will take place on ASICs. China can do it, even if the chips are not the most power-efficient. They have built lots of electricity capacity.
Ultimately however, that just means that models will become dirt-cheap. The money will be made with applications built on top of the models.
Fun as the asic play would be it has a giant hole - context storage. Raw speed only gets you so far if you can’t store and cache
I think the SotA is moving too fast for the production timelines of an ASIC, wouldn't you think? People are just now coming out with LLAMA ASICS but who would want to use LLAMA? Or I guess you are arguing that the models _now_ will be durably useful enough to commit the time to creating the ASIC?
> the winner will be whoever burns their models to ASICs fastest.
I don't understand how this works when the models are evolving so fast that your burned ASIC is outdated (or at least not top of the line) in a few weeks.
How does an ASIC manage to have 60x the memory bandwidth needed to achieve that speedup?
why not start with fogas ? you know … to test the waters ?
GPUs are already ASICs.
Apple's M7 comes to mind...
Another reason this may be true, selling hardware may be the only durable moat after a while
Open weights are a red herring if you can’t run the model independently. Good luck fitting these models on your Mac mini.
In fact the superior models are irreducibly nothing but superior web services run from China.
What precisely is an ASIC supposed to do that a more programmable accelerator can't? Memory latency is memory latency: doesn't matter whether it's embedded in a cache-line wait state or some flipflop state machine. ROM isn't going to be faster than RAM either.
Likewise, for compute, is the ASIC somehow going to beat a systolic array? You can't have one circuit per weight: the die area and electrical fan-out would be insane. I'm not seeing how an ASIC specialized for a specific model would actually help much.
I mean, sure, we can build more specific accelerators, e.g. for softmax, but these work fine in the context of a programmable pipeline.
Yes, there are more exotic things out there, like optical matrix multiplication systems. Those are different. But above, aren't you talking about just doing conventional digital linear algebra, but with a model-specific set of circuits?
[dead]
I'm not convinced, mostly because things like crypto, which I believe went into ASICs, were based on very slowly moving and mostly understood algorithms. LLMs and model architectures seems significantly more volatile. I wouldn't want to be working out the finer details of my chip rollout only to find a new paper/approach that give multiples of performance.
So I guess it depends on how much the latest-greatest model motivates people, and my read on the current churn is that developers are extremely unloyal to brand at this point and will jump to whoever has the best model. And as long as the best model is running on programmable GPUs, that will be the dominant form.