You should have set up https://github.com/mozilla/sccache at minimum. Setting up compiler caching is routine for people coming from older compiled language backgrounds but it's often missed by people coming from languages like JS/TS who have never been exposed to these ideas.

There's a new project called kache https://github.com/kunobi-ninja/kache that takes this even further and handles more edge cases with worktrees, but it's newer.

If you were pulling the entire project fresh and rebuilding everything in every worktree all the time without any caching then it would be frustrating. You could have asked your agent to set up basic caching and solved most of your problem in minutes.