I've had similar concerns with letting agents view any credentials, or logs which could include sensitive data.

Which has left me feeling torn between two worlds. I use agents to assist me in writing and reviewing code. But when I am troubleshooting a production issue, I am not using agents. Now troubleshooting to me feels slow and tedious compared to developing.

I've solved this in my homelab by building a service which does three main things: 1. exposes tools to agents via MCP (e.g. 'fetch errors and metrics in the last 15min') 2. coordinates storage/retrieval of credentials from a Vault (e.g. DataDog API Key) 3. sanitizes logs/traces returned (e.g. secrets, PII, network topology details, etc.) and passes back a tokenized substitution

This sets up a trust boundary between the agent and production data. The agent never sees credentials or other sensitive data. But from the sanitized data, an agent is still very helpful in uncovering error patterns and then root causing them from the source code. It works well!

I'm actively re-writing this as a production-grade service. If this is interesting to you or anyone else in this thread, you can sign up for updates here: https://ferrex.dev/ (marketing is not my strength, I fear!).

Generally how are others dealing with the tension between agents for development, but more 'manual' processes for troubleshooting production issues? Are folks similarly adopting strict gates around what credentials/data they let agents see, or are they adopting a more 'YOLO' disposition? I imagine the answer might have to do with your org's maturity, but I am curious!