logoalt Hacker News

TacticalCoderyesterday at 11:04 PM0 repliesview on HN

> I think people really downplay the impact of memory issues.

I don't think so. I've got my important data backed up on offline HDDs and SSDs and my most important data lives on a RAID system (ZFS) on a Xeon with ECC RAM but...

... there are simply a huge lot of checks everywhere making it so that bit flips on non ECC systems simply aren't that serious.

We live in a world of checksums and retries at every layer and distributed systems that are just about everywhere.

I went on for decades without ECC memory and nothing bad happened. Yes, I had the occasional bitflips. I did found out faulty mem using memtest (or weirder stuff, like some RAM stick identical to the other three, same chip everywhere, same sticker: but reported by memstick as being some other mem).

But I don't think that the "one bitflip cascading into corruption of your life's data" is something necessary to freak about.

It's not happening. Never happened to anyone.

If you (or other) are really concerned, you can use my scheme:

    - dsc09874.jpg
becomes:

    - dsc09874-b3-73b0835221.jpg
which means "that file's Blake3 checksum begins with 73b0835221" (use something else than Blake3 if so desired).

About 1/3rd of all my files are "tagged" this way, directly in the filename: this survive different medium and different OSes.

I then have my custom rsync backuping script doing a dry-run and warning me in a "stop the world!" fashion if it detects a bogus checksum (either on the source or on the dest: during the dry-run any modification to a file with a checksum is verified and any checksum mismatch means "stop the world").

I then also have, in addition to my custom backuping script, another script that allows me to verify either all files with that naming convention or a percentage I pick of those files (so basically some random sampling).

It's not mutually exclusive with ZFS / a system with ECC RAM.

> but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time

I hear you but... How much do you really need, years later, a file that you haven't needed for years? That one file is unlikely to be a big loss.

Now I'm not saying you're wrong: I ended up with my naming scheme (appending a partial cryptographic hash to the filename) after finding a series of about 20 pictures that were corrupted. And on all the backups I first checked, they were also corrupted. I ended up finding a very old data CD on which that series of .jpg files were fine. And I decided "never again, I want such SNAFU to be detected in the future".

But truth is: with 40 000 pictures, life without those 20 pictures would still be fine.