I think we need to just think of agents as people. The same principles around how we authenticate, authorize and revoke permissions to people should apply to agents. We don't leave the server room door open for users to type commands into physical machines for good reason, and so we shouldn't be doing the same with agents, unless fully sandboxed or the blast radius of malign or erroneous action is fully accepted.