DskDitto: Ultra-fast, parallel duplicate-file detector

Wow. Wasn’t expecting to see this here… I am the author…

Why do you need a cryptographic hash function for detecting duplicates? xxHash, rapidHash or anything that’s fast will do the job. There’s no need to be secure against inverses.

I'm not sure exactly how it compares, but a similar tool you can use that also does phashes of images/videos to find potential duplicate media files is czkawka [0].

[0]: https://github.com/qarmin/czkawka

I have a thing like this - one thing I have found useful is also going up the directory tree to find entire duplicated directories. (So dir hash is hash of the concatenation of the subdirs and the files).

I also used go, it excels at just this sort of thing. Although I saturate the disk to memory bandwidth (for the SHA) way sooner than CPU on my laptop, so it still takes a while to run.

Nice.

Here's a tip, if you are making regular compressed backups of the configuration of your network devices, if you use gzip, make sure to set the mtime to 0 when writing the file as this can throw off deduplication. This also applies to the name field.

https://docs.python.org/3/library/gzip.html#gzip.GzipFile.mt...

Reflink conversion would be another nice feature to have. It allows userspace apps to create filesystem-native copy-on-write clones of files, so to the next app they work exactly like another copy of the same data. But the data is on disk only once.

> …finds duplicates across large disks instantly

:D

Interested to know what algorithm you use for --fuzzy. Presumably not any of the worst-case-quadratic diff algorithms... MinHash on n-grams?

imo fclones is the one to beat, https://github.com/pkolaczk/fclones#benchmarks

nice that there are actually some ok interfaces here. with fclones, i tend to generate a file of candidates then do a little review to make super certain everything is square / as expected. it's very built for intermediary files a core pattern, which is very unix, very convenient. but it did take me a little while to settle on this, and it felt like i wasn't being offered a ton of options for management out of box. looks like some real attempts to be more user friendly here.

[dead]