If you're really wanting to do perceptual hash based deduplication, use multiple, heterogeneous hash algorithms (phash, dct hash, mean hash, ...) as it is likely that a given hash algo will happily lossily match with very very different images--but if all hashes match, you're much less likely to have false positives.

I wrote up what I do here: https://photostructure.com/guide/what-do-you-mean-by-dedupli...