alt Hacker News6x. Taalas has Llama3.1 8B running at 18000 tok/s. Cerebras advertised that model at 3000 tok/s.
Replies
HNisCIS • last Monday at 5:49 PM
8B is still three orders smaller than current frontier though
6x. Taalas has Llama3.1 8B running at 18000 tok/s. Cerebras advertised that model at 3000 tok/s.
8B is still three orders smaller than current frontier though
Interesting. Is the speedup from specializing for the shape of Llama 3.1 or are they (contra my mental model) actually winning on burning in the weights?