Absolutely - one of the things I was testing with the harness was free reign to install packages, modify the system, etc. Basically an anti-harness.

In my early testing with 5.5, I didn't see this behavior, so I didn't lock down the sandbox.

For the vanilla Codex runs, I just used the benchmark's built-in Codex package, so it's not clear to me if the published benchmarks have access to the internet or not.

If I were to continue benchmarking, I would allowlist certain package repository URLs, instruct the agent not to cheat, etc.

As noted at the bottom of the post, Terminal Bench 3.0 explicitly asks the agent not to cheat[0].

[0]https://github.com/harbor-framework/terminal-bench/blob/v3.0...