alt Hacker NewsFrom what I know about batch processing/ concurrency in inference this is a pipe dream... Or its going to cost an arm and a leg. I think they're lying or its going to be a much smaller model and not "frontier"
From what I know about batch processing/ concurrency in inference this is a pipe dream... Or its going to cost an arm and a leg. I think they're lying or its going to be a much smaller model and not "frontier"
You have speculative decoding that easily increases speed 2-4 times with no loss of quality, and of course MoA architectures that speed up inference 10 times or more, although with some quality loss.
Better hardware, and other techniques on top of that and you speed up even further.