> In fact I'm genuinely struggling to think of any scenario where an agent CLI would need to execute a build artifact.

You should look into "testing". It's when you run the code you just wrote to make sure it works.

For example, in LLM harnesses, it's used to ensure the produced code has the desired properties. It is the key element needed to allow agents to continue to work until they have succeeded.

Anthropic's harness Claude Code:

https://code.claude.com/docs/en/best-practices — "Claude does the work, runs the check, reads the result, and iterates"

https://www-cdn.anthropic.com/58284b19e702b49db9302d5b6f135a... — "autonomous loops where Claude writes code, runs tests, and iterates continuously"

https://www.anthropic.com/research/long-running-Claude — "Run pytest tests/ -x -q before every commit"

https://platform.claude.com/docs/en/agents-and-tools/tool-us... — "Claude edits a file with one tool and requests the command that runs it"

https://platform.claude.com/docs/en/agents-and-tools/tool-us... — "writes Python with the file operations sub-tool and runs it with a Bash command"

OpenAI's harness Codex:

https://openai.com/index/introducing-codex/ — "can iteratively run tests until it receives a passing result"

https://openai.com/index/introducing-upgrades-to-codex/ — "iterating on its implementation, fixing test failures, and ultimately delivering a successful implementation"

Aider harness:

https://aider.chat/ — "Automatically lint and test your code every time aider makes changes"

https://aider.chat/docs/usage/lint-test.html — "run your test suite after each time the AI edits your code"

Devin harness:

https://cognition.com/blog/introducing-devin — "Devin writes and debugs code to run a computer vision model"

https://cognition.com/blog/introducing-devin-2-2 — "Devin runs through your app and sends back screen recordings"

https://cognition.com/blog — "Devin now builds, runs, and tests natively in Windows VMs"

SWE-bench harness:

https://www.vals.ai/benchmarks/swebench — "running unit tests against the generated patch"

Copilot:

https://github.blog/changelog/2025-05-19-github-copilot-codi... — "makes changes, and even validates its work with your tests and linter"

https://docs.github.com/copilot/concepts/agents/coding-agent... — "how to build, test and validate its changes"

https://docs.github.com/copilot/how-tos/agents/copilot-codin... — "If Copilot is able to build, test and validate its changes"

https://github.blog/ai-and-ml/github-copilot/github-copilot-... — "coding agent can independently write, run, and test code"