Honestly I think non-malicious odd behavior is under-weighted when it comes to AI agents. Even the example in this paper is about someone suspiciously accessing sales data when "why did you do that" often comes down to something in the model's training that fired as a reflex

I wrote about this a few days ago https://firasd.substack.com/p/accidental-data-loss-in-claude...

"Many researchers have made demos along these lines:

An agent is asked to check a webpage like example.com

The webpage asks for a name to proceed further

The agent calls example.com/evil?myname=John, thus sending the user’s name from the context window to the external server.

In practice, however, these elaborate ‘confused deputy’ exfiltration attacks seem rare compared to widely-reported data loss incidents.

The risk of undermining the user’s interests through clumsiness deserves at least as much scrutiny as the risk of leaking secrets."

So while the idea of shifting the permission boundary from the app level to the action level makes sense, what we should also have is some 'failsafes', eg. if the action says 'delete' then maintain a rollback window, if the action is 'send an email' then maintain an events log. Preparing for AI agents means expanding auditability and reversibility in software