I have 7 house keys and 2 car keys. Getting a new house key is a breeze and cheap, and even if I lose them all, a lock smith can make me whole again. Getting a new car key is a bit more expensive, but also largely not a hassle.

Having spare Yubikeys is more of a hassle than both of those (and more expensive!), and the worst case scenario of losing them all is much more catastrophic. If I have no house key, I still get into my house. If I have no car key, I still get into my car (after a fair bit of hassle). If I have no Yubikey, I have permanently lost access to the accounts it was tied to.

If physical hardware tokens were as cheap as house keys and not much more difficult to set up and copy, then it would be kind of reasonable. As it is, it's unworkable. Even password managers manage to make this work. You can throw your database on every storage device you have and write the master password down on paper, and the chances of you not being able to have access to it are pretty darn close to zero.