Nothing stops it, except the fact that the agent is already executing arditrary code in your shell. If it's malicious, it'll just steal your shh keys directly instead of bothering with button masking