Yubikeys (or something like keys in the Secure Enclave of a Mac or iPhone) are themselves not accessible by software, you can only ask the hardware to perform operations with them (e.g. decrypt or sign something).

This means someone with access to your filesystem can’t just copy the key off.

Of course a passphrase is another protective layer. But passphrases can be keylogged, or the decrypted key (or passphrase, depending) in the memory of an ssh agent can be can potentially be read out by an attacker.

With separate hardware, the assumption is that the hardware is much much harder to attack, so the keys stay on there and only on there.