alt Hacker NewsI tried Cerebras this week for my project and their 27b model at 2000tks would be perfect for it. Unfortunately it doesn't do the 2k tokens per second which is what I am looking for.
Replies
villgax • today at 7:30 AM
And the fact that you will never be benefited from the batching they’ve explained either, you’ll be paying for every token sequential or parallel unlike nvidia or amd gpus
are you evaluating using OpenRouter.ai or other routers/gateways?
they have never been able to keep up with Groq/Cerebras-level inference speeds for some reason.