Let me make sure I'm getting this: They asked the models for answers but also for confidence score.
Jev's confidence score was more reliable. Which means that they were able to use Jev despite it having lower accuracy, because of the confidence was below a cut off, they would escalate to an expensive model (Opus)?
Whereas Luna, even though it was more accurate than Jev, its confidence was way off, so they couldn't reliably know when to fall back?
It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.
- Mark Twain