I don't think we require a Trusted Publishing provider to use a physical HSM. As I've said in other venues, even a self-hosted Vault instance would almost certainly be fine. But storing private keys on disk is not, and we owe PyPI's users a higher standard of care than saying otherwise.
(I also don't hand out gold stars, and my employer does not direct my work on PyPI.)