alt Hacker NewsI have a thing like this - one thing I have found useful is also going up the directory tree to find entire duplicated directories. (So dir hash is hash of the concatenation of the subdirs and the files).
I also used go, it excels at just this sort of thing. Although I saturate the disk to memory bandwidth (for the SHA) way sooner than CPU on my laptop, so it still takes a while to run.
Nice.
I wrote a tool a short bit ago to handle massive parallel hashing. Originally, it was a testbed for a project I built to make multiprocessing easier in Go.
https://github.com/indrora/hyperhash
In it, I learned that there's some interesting quirks about the way the traditional shaXsum tools work, specifically that they focus on a set of inputs from the shell command line, which means that you can exhaust the shell command line length fairly easily (try hashing every file in the Linux source tree, for instance; zsh, bash, and fish will all eventually tell you "Fuck off, there's too many characters in argv")
I also discovered that Go's handling of file handles varies drastically based on OS. Windows, for instance, has no problem with you taking a file handle to every single file on a disk. Linux will get upset but eventually capitulate, macos will stomp its feet and inform you that you are a bad child and kill you off.