The thing I find burdensome is managing all the keys in a secure way. I think I would want a hardware token to store the keys on, have separate keys for every site, need to back up my keys onto a second token in case the first one is lost, etc. It gets burdensome. At least with passwords you can store them in various ways that are not hardware or software dependent.