I know Syncthing is great about verifying the data read from device A matches the data received at device B, but as I understand it, it doesn’t guarantee the data written to device B was written correctly.
If it wasn’t written correctly, I assume the file timestamps and/or the recorded hash that Syncthing keeps would prevent Syncthing from noticing unless I did something, like a rescan? And then watch to see if any content was transferred?
Thanks.
IMHO, best practice is a server with ECC memory and a file system that keeps block hashes, e.g., zfs.
It would be a great option if syncthing exposed an option on a folder by folder or device basis to verify writes. I.e. before renaming the file into its final filename and committing the database if the temporary file was rehashed and verified it matches the global hash.
Given all of consistency checks all through the process it’s an interesting choice to omit final verification that the file is correct on disk.
Absolutely you can say it’s the responsibility of the OS and file system to assure integrity of the file written to disk but I suspect the vast majority of systems running syncthing are not using ZFS and ECC memory.
Is this something that could be considered?
One complication I see is that it could give users a false sense of security because there’s no guarantee that once Syncthing has verified the hash of each data block, it stays that way moments later. Then when it’s discovered that files are corrupted, the blame will unfortunately be directed at Syncthing.
If ECC memory and a checksum-capable filesystem isn’t an option, there are many tools for bulk generating and cataloging file hashes.
Bit rot story…
Years ago I had an external USB HDD packed with audio files encoded in FLAC format. The ext3 filesystem didn’t report any errors, but I happened to notice some unusual quiet “popping” noises during playback of a track.
Opening the suspect file in an audio editor, I saw unexpected spikes in the waveform. When I checked the MD5 signature embedded in the metadata, it didn’t match.
After finding hundreds of corrupted files, I ran a media scan for bad blocks, which turned up thousands of them. I don’t recall the exact results, but more than 20% of my music files were corrupted without any warning from fsck.
Although I hadn’t backed up the FLAC files, I had the original CDs, so it was a very tedious lesson re-ripping them.
Ever since then, I’ve used Btrfs whenever possible on portable storage plus a mix of Btrfs and ZFS on servers (and also greatly improved my backup regimen).
I’ve also found bit rot on some of my writable CDs/DVDs due to degradation of the organic dye and/or oxidation of the aluminum reflective layer – the lacquer coating sprayed onto the label side isn’t always perfectly impermeable to moisture and air (so far the gold discs haven’t had any issues).
Precisely this, imho. We verify the data hash before we send it to the filesystem. After that, really, there are existing solutions that do it correctly. A single readback doesn’t say very much because it will presumably come from cache.
Or, you can use a regular userspace hash checking tool like SFV files.
I hear you but by that logic you guys should delete versioning because it may give people a false sense that syncthing is a backup program.
Anyway I don’t think this “false sense of security” reason is a valid reason not to verify data. Of course syncthing isn’t responsible for bit rot and bit rot is real and well documented.
The implementation problem of reading back from disk instead of from cache is a valid concern. i suspect there are ways to read back from disk but different platforms probably have different mechanisms and that gets complicated.
Anyway thanks for listening.
Just to be clear since this sort of makes it sound like Syncthing doesn’t do due diligence. All data is verified against sha256 when read, transferred and written.
What you’re proposing (to read back data just written to somehow try to catch hardware failure) is something I’ve never come across as a suggested practice. Are you aware of any application that does this? And, as mentioned, for those who care (I’m one of them) there are solutions in place for doing this properly - much better than we ever could.
I mean the idea of verifying writes goes way back…
How about xcopy as an example.
I’m not suggesting syncthing isn’t doing due diligence. It’s just doing lots of due diligence up until the point of handing the data off to the filesystem.
Unless xcopy’s code base has been greatly enhanced after Windows 8.x, its post-write verification simply compares file sizes. If it weren’t for backward compatibility considerations, Microsoft might have removed the option.
Note that rsync also doesn’t do a post-write hash verification – its rolling checksum validates data chunks in transit, then leaves it to the OS and hardware to take care of the rest (e.g. ECC in RAM, HDDs and SSDs).
I’m the OP not the one you were responding to… Can’t say it’s anyone else’s SOP, but When I’ve manually synced things I’ve often used a diff tool (Beyond Compare) to sync, then did a binary diff at the end to confirm there were no issues. The diff I would do after pretty large copy operations, so I assumed caching wouldn’t be an issue.
A handful of times over a decade or so I have found issues with minor corruption of individual files. Now, in my case I was copying by different means (SFTP, MTP, SMB, USB mass storage) and I don’t know what mechanisms of protection each method included to reject corruption in transit… and I don’t know where the corruption occurred, in transit or on disk.
I understand that Syncthing shouldn’t take over the filesystem’s or OS’s role, but it just seemed like it’d be useful to know that at one instant in time the data on the destination was known to be identical.
And I guess I go back to my original question… Even if Syncthing wasn’t meant for this, can it do it, if I force a rescan and then see what (if anything) it needed to re-sync? (Apologies if I’m getting the verbiage wrong.)
Thanks to everyone for the interesting discussion on this topic.
Rescan doesn’t read each file back. It only checks metadata to look for differences. (File size, modified date) If I’m correct you’d have to delete the database and resync for the files to all be rehashed.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.