alt Hacker NewsOpenRouter is great business I agree, but I'm not still convinced how the integrity of providers' models is ensured. In other words, can't the provider serve DSv4 flash advertising it as DSv4 Pro?
I'm aware that OpenRouter checks response quality onboarding, and does further checks occasionally, but I'm concerned that it's basically a cat-and-a-mouse problem between the scammers and the detectors. For example, there could be a signal that a specific pattern of requests are from OpenRouter's quality testing bots. Or, they can just route 1% of requests to an inferior model and benefit a small gain, hoping it fits into the statistically allowed margin.
Replies
Yes, it is awfully inconsistent today. They run some tests (accuracy table buried on model page) but they are sparse and only capture a single moment in time. I would love to see OpenRouter take this more seriously.
How is that a problem that's unique to openrouter? Who's to say when I query the Anthropic API directly, wanting to use Opus, but they determine it doesn' need Opus, route it to Sonnet instead and pocket the difference? The only difference with openrouter is that it's another layer where this type of fraud could occur.