alt Hacker NewsAccording to the benchmarks here they're claiming up to 97% accuracy. That ought to be good enough to trust them right?
Or maybe these benchmarks are all wrong
Replies
Gemini routinely makes up stuff about BigQuery’s workings. “It’s poorly documented”. Well, read the open source code, reason it out.
Makes you wonder what 97% is worth. Would we accept a different service with only 97% availability, and all downtime during lunch break?
Does code work if it's 97% correct?
It's not okay if claims are totally made up 1/30 times
Of course people aren't always correct either, but we're able to operate on levels of confidence. We're also able to weight others' statements as more or less likely to be correct based on what we know about them
> Or maybe these benchmarks are all wrong
You must be new to LLM benchmarks.
Something that is 97% accurate is wrong 3% of the time, so pointing out that it has gotten something wrong does not contradict 97% accuracy in the slightest.