Yeah, if you don't care if your ssh keys or data are potentially posted all over the internet, more power to you...
Not to mention cases like some comment here:
"I've caught Fable discovering the ip to a production server in documentation and attempting to connect there on its own to run commands without explicitly being prompted to. It didn't work because I was watching it live and and also the key was password protected, but yeah, I do see some danger."
How are ssh keys going to be leaked, the agent never needs to read them.