logoalt Hacker News

guanlan • today at 2:57 PM • 2 replies • view on HN

Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.

The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x

What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.

What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.


Replies

radlad • today at 3:49 PM

Hi Guanlan, I vouched for your comment since you involved in this post, but you should be aware that all of your posts have been flagged, likely for AI patterns in your writing.

> Don't post generated text or AI-edited text. HN is for conversation between humans.

https://news.ycombinator.com/newsguidelines.html

latentsea • today at 4:08 PM

This is why for open-weights models at least harness aware post-training is becoming a thing. Also closed source harnesses that aren't fully editable are basically an anti-pattern. The labs should just hurry up and productionize self-evolving harnesses, and have the models internalize harness engineering, so they can optimise the harness for the given task. These are solvable problems.