At least my password won't leak as often with yubikey, but the attacker can still hack my shell to execute fake sudo. Even if I type /bin/sudo explicitly, there is ptrace, LD_PRELOAD or just replacing the entire bash binary.

In practice yubikey sudo keeps you much safer today, as almost nobody uses it and malware won't be prepared for it