Yes: if you use a Trusted Publisher with PyPI, you can constrain it to an environment. Then, on GitHub, you can configure that environment with a tag or branch protection rule that only allows the environment to be activated if the ref matches. You can also configure required approvers on the environment, to prevent anyone except your account (and potentially other maintainers you’d like) from activating the environment.
If they have compromised the token wouldn't that mean the developer is compromised and such access can be used to just put "curl whatever" into the build and publish that payload on pypi?