This is an interesting problem. A similar approach that worked for us was to only load the tools the agent actually needs for each task.