logoalt Hacker News

bisonbear • today at 5:13 PM • 1 reply • view on HN

In this case, I am not entirely sure how cost per pass is distinct from cost per successful task given that success is defined as pass/fail.

To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode


Replies

Edward40 • today at 6:35 PM

We use cost per pass, not average cost per task, because we want to see the harness keep a high pass rate and low cost at the same time.