For us, it would be for SRE stuff. We have agents reviewing traces and logs, and inspecting system behavior daily - for non obvious problems, not surfacing in metrics. When we hit an issue, we use the /fast mode to triage, propose a fix and then build and deploy. It is trivial amount of money all things considered, and I'd happily authorize a 100x spend for when we have a prod outage on a mission critical service.

When you think about it, it would still be dirt cheap compared to normal way of doing things. In the old days, if you had an outage on a serious user facing system, you'd wake up people across various timezones, wake up their managers and scramble to find the root cause, identify a solution, brainstorm on possible side effects of a fix, and then rush to build it and deploy. This cycle would involve, sometimes, dozens of people, for, say, 10 man hours each. So lets make it 120 man hours per serious outage, and lets assume and average of $100 per hour - so, $12,000 per a serious outage fixed under a day, counting conservatively and not including the costs of the actual outage.

I'd guess the pricing for those ultrafast, very energy inefficient and hardware heavy models will be competing with that. Its going to be possible to get a fix out in 30 minutes, 10 of which will be tests, 5 will be the deploy, and the remaining 15 will be some unlucky guy trying to keep up with the super fast model throwing a 50 "load-bearing deferrals earning their keep" per minute :-)

The pricing on those things is competing with costs to run entire departments. I'd, for one, imagine offshore ops teams will be a thing of the past in under a year, since one gets way better initial response to anything from a model, given right setup, esp. on codebases that have been built from the ground up with agentic coding - so with good documentation and effective test coverage baked into repos.

There are many industries and situations that will pay absurdly high multiples on API rates for low latency access. Anything remotely related to finance, deal flow, or trading, including the development and checking of Excel financial models. Live analytics that can be delivered mid-meeting to C-suite executives, or ultra-luxe dealmakers who need answers while on a live call. Intelligence and defense, where milliseconds matter.

With the tool calls that can be done, you're not pricing this against an executive assistant or pocket analyst - you're pricing this against the ability to have an entire Bourne Identity style analysis room at your disposal. The limited inventory will go to the people for whom money is no object.

I have no idea, but I've assumed that batching can't work on Cerebras.

Batching works because of severe memory bottleneck, but Cerebras whole thing is serving models out of "L1 cache" (?).

You'd need hundreds of GB alone for the KV cache of each user. For something like LLama 3 405B you need ~67GB at ~130k tokens. A single CS-3 has 44GB on-chip sram.

So, afaik, Cerebras are optimizing for ultra-low latency batch=1 inference.

https://newsletter.semianalysis.com/p/cerebras-faster-tokens... goes quite in-depth.

They're cache limited, almost certainly, so more slower sessions doesn't solve the problem - you still have to load and unload the whole cache hierarchy at some level and that's a network bandwidth and memory bandwidth problem between the external systems and the waferscale chip.

Also worth looking into how they do cooling for it, because that's kind of absurd and awesome as well.

> I don't think I understand why they aren't leveraging the increased speed to do batching to serve more customers at a "normal" tok/s.

There's some technical hypotheses about it that other people are offering.

But also from a business perspective, it totally makes sense not to go any sort of batching play. It's really valuable and very clear to consumers to make your pitch entirely about lower latency rather than higher bandwidth.

There are so many scenarios that are latency-constrained that will be difficult or even impossible for someone even with fleets of high-bandwidth compute to compete with you on.

Very easy pitch to sell a customer who asks what differentiates you from other companies: you pay us a premium for lower latency than anyone else.

Cerebras is a large plate sized chip. It has 50GB of SRAM, and few hundred K simple cores that can access that SRAM really fast. I don't know semiconductors well, but I understand that the same manufacturing technique that makes this huge chip possible, on the flip-side limits inter-chip communcation bandwidth. In cerebras, it is 150 GB/s (compared to nvlink's 2TB/s or groq's similar).

One way large models are served on a bunch of cerebras chips is by essentially distributing layers' weights across chips. Few layers's weights per chip - as many as the KV cache + activations + weights will allow. You use pipelining to hide the latency of the inter-chip 150 GB/s link.

On GPUs, you amortize the cost of loading weights from HBM to SRAM across multiple users - thereby making it cheaper _per_ user. But here, there is no such amortization. The weights are already there. It is the activations that stream through.

You _could_ do batching/continuous batching, but that would just service more users at lower token/s each without any amortization of fixed cost, due to fixed cost (loading weights) being non-existent.

To use a car analogy, they built. Ferrari and technically you could put a tow hitch on it and go slow and carry a bunch of cargo instead of 1 passenger and go real fast, that's just not what they've built. There's one really fast core to send tokens at and get tokens back out at insane speeds.

Because companies are willing to pay proportional to (tokens/s)^k for some value of k >1