<< that would severely limit the kinds of tasks you can use the agent for.

Are we just talking about API calls to providers? If so, wouldn't local agent + sandbox solve all that?