alt Hacker NewsThe benchmark wasn't evaluated with the router, we wanted an apples-to-apples comparison of our harness against other harnesses like mini-swe-agent on the same models to see if we added speed + cost value beyond routing.
The benchmark wasn't evaluated with the router, we wanted an apples-to-apples comparison of our harness against other harnesses like mini-swe-agent on the same models to see if we added speed + cost value beyond routing.
Okay, that makes more sense, but then this post is very confusing. Why mention the router at all? Or does your live system have a router and the eval doesn't? In which case why not publish an eval with the router?