The standard way you distinguish "adversary with physical possession" from "owner" is "do they have the owner's passphrase".