Because these tasks are genuinely hard and require long-horizon reasoning, running full evaluation experiments gets expensive fast.
For example, on bun-sourcemap-leak (https://hub.harborframework.com/tasks/terminal-bench/bun-sou...), all 440 trials finished cleanly with zero infra errors or timeouts, yet 0 out of 440 passed all hidden-variant tests (the best was 35/36 across three DeepSeek V4.1 Flash runs).
Some of these tasks take up to 8 hours per run, so if you are not careful, you end up burning massive amounts of compute and API spend just to get an accurate evaluation signal.
I wouldn't call it "burning of tokens and compute" if you get a useful signal out of it.