alt Hacker NewsGoogle's research showed some of the speed benefits tended to drop off at larger batch sizes on an H100. Are there published batch-size vs tokens/s figures for Mercury 3?
Google's research showed some of the speed benefits tended to drop off at larger batch sizes on an H100. Are there published batch-size vs tokens/s figures for Mercury 3?
We've been happy with how performance has scaled with batch size. The model runs in our API at reasonable costs, which is also what allows us to serve the model with competitive prices ($0.10 per 1M input and $0.35 per 1M output at launch).