I tend to agree that there are issues in the realm of hardware token based security but they largely lie in the failure modes, with substantially increased difficulty of recovery.

For the average user these approaches make data loss MUCH more likely simply because you need a corporate IT department level of competency to consistently avoid data loss with them.

The glib exaggeration of this is that in not being permitted to manage and back up your own keys you actually create the situation where you have to hire someone to extract your keys for you and break into the device you own because of the failings of the technology!