One annoying thing having a guard model is ensuring its FP rate is low enough to not be a headache, and its FN rate low enough to be worthwhile to use. Likely also means you’ll need to either use a frontier-close model, further increasing costs as well as round-trip time.

Or maybe some system 1 model?

For my setup, i have claude code in its own VM, whose VLAN is heavily restricted outside of the internet, my forgejo instance, infisical for secrets, and one way SSH from my machine to the VM.

All secrets are shared via infisical secrets store, so the actual secrets are never shown to the agent. Dev mode can be used for running the system locally during development, and those secrets may be seen in some cases. The foregjo user the agent has access to can only create issues and PRs. Manual merge required, after which a CI pipeline handles the packaging and “production” deployment of the app to my app server VM.

I think a setup like that (or discouraging these agents access to the same computer people keep their finances/health data etc) should be discussed/championed more.

My work encourages just running the harness on their system (which I get for practical reasons among others), to which I then put my work laptop on its own VLAN.

Overkill, for sure. But to me the risk of personal data exfiltration is far too high with these vibe coded harnesses and models which are more and more opaque.