If your threat model is that of a malicious agent that will use a 0-day LPE to get root and exfiltrate all of your SSH keys, virtualization makes sense. But then, I wouldn't run such an agent at all, if not specifically in the context of malware analysis.

If you're just concerned about "agent messing up and taking the rules in some markdown files more laxly than I would have", then running it as a seperate user is totally enough...

What kind of things are you even doing that the agent would try to perform a kernel exploit on you? I thought sandboxing is just to protect from the agent accidentally clearing your home directory.

Side note, just 6 days ago a Linux VM escape exploit was disclosed.