You can require physical user interaction to use the key. If the mossad is after you, this doesn't really help you because they'll just replace the ssh binary and intercept one of your requests, but against a less motivated attacker, it makes it harder to pivot from one machine into others.