alt Hacker NewsNot sure how good the Harvey benchmark is. I recently looked into traces from an agent solving their firm knowledge task subset and found multiple problematic tasks and rubrics.
Here are 3 examples (there are more): https://github.com/harveyai/harvey-labs/issues/191
Replies
Bolwin • today at 5:44 PM
This is why I severely dislike using all pass rate. A model could be at like 93%, meeting practically every requirement except the broken ones and all pass would give it like 20%.
Harvey bench is probably one of the worse benchmarks out there. Agents as judges, vast scope, some questionable problem/solution pairs. Tricky domain TBF.