alt Hacker NewsAs an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.
Replies
It is amusing how many people are arguing what litigation is here, probably none of these people have a law education.
Imagine a bunch of lawyers arguing what programming is. Even a bunch of programmers arguing what programming is abnoxious at worst and a mental exercise at best.
AI is coming for the transactional work long before it will come for litigation. In 50 years, nobody is going to have a human write their will, for instance.
Objectively a 20% pass rate is not impressive. Yet, there is still utility to AI research and drafting.
If professional AI can do all the work that isn’t novel, you free up human talent for edge cases and then improve your AI with new information.
How much of your work is truly novel and unique? Not 100%
What are you talking about? 90% of litigation is convincing the judge that existing case law that clearly applies to your case does apply to your case.
[flagged]
How much novelty can there be? If attorneys were as diligent and generous with “open sourcing” their data as SWEs had been, could LLMs benefit from all that training data?
To be fair to them, on this particular benchmark (Harvey's Legal Agent) 20% puts it almost top of the pack of _any_ model, and certainly top for any remotely in its price class.
Opus 5.5 scores 4% on it.