Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it
it only cost them $10,000 in Claude API usage. :)
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it