There is also a middle ground - it's possible to sign SSH certificate using yubikey as CA. By controlling expiration and set of principals, the cert you get on your machine is scope restricted, and quickly expires. Or you can load it into an ssh-agent and drop it immediately.