alt Hacker NewsMy paranoia tells me they A/B test users which means consistent benchmarks go through the high performance pipeline while regular requests fall back to the quantized / experimental / low cost version. I can see the main reason being the oldest one in the book, which is making more money.
Replies
eithed • today at 9:08 AM
It's not paranoia and would align with https://www.reddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
I wonder as well whether would this pop up on nerfbench
kasey_junk • today at 11:50 AM
Do you ever a/b test them? Run the same setup and prompt multiple times to see the output?
It could also be a regional effect where systems are under higher load and hence dynamically routing to / provisioning lower precision instances; or some other factor where one is affected whereas someone else isn't.