Hyperspace

It's quite the dark-pattern to allow you to download it and scan for free, without any clear indication that it's a paid product, and then ambush you with a purchase dangling the space savings in front of your face.

For Windows users looking to save space, there's Compactor[1] that uses the NTFS built-in compression opportunistically for only those files that benefit from it. It is not a deduplicator but the basic idea is the same: take an underutilized OS feature and make it usable.

Pretty handy when your flight sim game takes 1.2 terabytes with most of that being sparse terrain mesh data and uncompressed textures. Compressing the whole directory would take over a day according to Microsoft and their infamous progress bar; Compactor gets it done in fifteen minutes.

[1] https://github.com/Freaky/Compactor

I think the record for most space saved by this utility in a single run is in the hundreds of TBs now. I’ll see if I can find the toot.

Edit: maybe a bit hyperbolic of me, looks like it was 3.94TB https://mastodon.social/@WTL/116710030179809319

It's amazing that this is a product. In an earlier job we used to ask how to do this (discovery phase of deduplication) in the first interview screening. Once you have that list, it's straightforward to make a few syscalls that make it happen.

This uses reflinks, right? I've been experimenting with using reflinks under Linux to speed up layer extraction for Docker, it's great.

How does this compare to diskDedupe, which has been around longer and is much cheaper?

Nice I was just thinking about this the other day! Given the memory supply issue today, I wonder how much of data in our data centers worldwide is essentially just copied data? I have a feeling that there is a ton of redundancy, much of it absolutely necessary, but much of it essentially not at all, and howmuc memory we can reclaim by culling copies

By comparison, have to handle it to DeDupe for its transparency:

Key Features Free to Scan, Unlock to Deduplicate

Surprised that app with Full Disk Access permission requirement is allowed to the App Store.

i asked claude to make this and in 15min had a working replica CLI

How does it compare to https://diskdedupe.com/?

look ma, antibackup!

Are you deleting the others and creating a symlink to the original? How does this work exactly? I didn't get it from the FAQ section.

Does this mean that if the original is gone, all the file links will not be found?

Seems useful for s3 buckets. I suppose a script that watches for new objects, calculates its sha256 and stores that in a DB, then checks for duplicate hashes would be a fairly trivial task. Though s3 doesn’t support symbolic links so accounting would need to be handled by server side code.

[dead]