I think the only solution to this kind of challenge is forcing the agent to go through a proxy which handles all the authentication and authorization for the agent (thus it never has too much access to abuse), and monitors for exfiltration or prompt injections.