Okay, I wanted to know “which tasks” and dig into the methodology, and its documented nowhere!
A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.
For real, i remember setting up advanced Linux configurations back before the 3.0 kernel and I'm pretty sure I had a higher failure rate.
The biggest difference is that I, as a human, get mentally exhausted. but modern AI (when given unlimited tokens and tools) is like if Stephen hawking had a meth habit and the tenacity of Sisyphus with a bad case of syphilis after being told he can be cured if he can get that boulder over the hill
And even if they did try to test them on a lot of gotcha tasks; I'm perennially reminded of the ever-relevant quote about AI and other tech: "this is the worst it will ever be"
I'm also building a leaderboard in the meantime, which will release in a few days with more detailed harness version info and methodology.
Confirmed. This is 17 TB4 tasks and 45 TB2.1 tasks. Why?
Sorry for the inconvenience. We ran 27,280 total trials on Runta (1–4 vCPU, 2–8 GB RAM, 32 GB disk) using the 62-task [email protected] dataset across 5 models and 22 harnesses on Baseten, with 4 attempts per task. Every harness and model ran at max effort with a 131,072 max output token limit per request and per-task agent timeouts ranging from 15 minutes to 8 hours.
For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.
https://hub.harborframework.com/datasets/frontier-harness-ev...