Since we got transferred files with content of only Hex 00 (NUL) characters (with same filesize as original file) we changed from version 1.4.0 to version 1.20.1. But we still encounter such files. We are using Syncthing for transferring files between two Windows 2019 Servers. Is there any known issue about that? Is it possibly a question of performance (we synchronize (bidirectional) about 15000 to 20000 files daily, with sizes of about 2KB to 15 MB)?
Syncthing hashes the file contents on sending and receiving side and compares those hashes during transfer, so this seems surprising. I’ve never heard of it, at least. Do you have files which are supposedly in sync where you compare the data on each side and it differs?
Yes, we have examples. Because of the error/corrupt transfers we ‘keep some file copies’ (where a real time transfer is not important) on the outside server. I try to upload the two exapmles (first file OK (extern), second file corrupt (after Synchthing transfer). 65002E77-F5EC-4D57-A0F4-B81FD6289D4B.xml (1.8 KB) 65002E77-F5EC-4D57-A0F4-B81FD6289D4B.xml (1.8 KB)
Interesting! Can you reproduce it at will somehow? Any patterns to which files it hits? Any odd things about the system(s) (unusual filesystems or filesystem filter things like antivirus)?
Do you use file watcher or interval rescan?
Also make sure the files are fully written before Syncthing starts transfer. It could be a race condition
Unfortunately there is no chance to reproduce this case. If we send the file one more time it is transferred correctly. Within the last two months we counted about 150 such cases, while transferring about 600’000 Files (still enough to lose confidence into the solution). And unfortunately we did not find any ‘irregularities’ on our servers (we also checked the event logs) nor specific day times. My be interesting: there were some very few cases (about 4, I upload 2 examples) where only a part of the files are written with HEX 0 (NUL) values. In these cases there where exact 128KB blocks with such characters. VE-VeranlagungsdetailsOrdentlich_20220513_090912_24fd5228.xml (641.9 KB) S_20220512_140819_e164f97fa6be42a59027d14371764a59.xml (1.1 MB)
What application writes those files? Something somewhat similar happens on android devices, where the system creates an empty file and then quite slowly starts adding the content. Thus it sometimes happens that Syncthing will scan it before the content is there. So if e.g. here a program creates the empty file, then waits for long enough for the filesystem watch to notice the new file and thus Syncthing scanning it, and then only later writes the actual content without changing the modification time Syncthing will never pick up the actual content. To check that next time this happens, bump the modification time on the device where the file originated. Syncthing should then rescan and propagate the correct content. If that’s the case you can increase the filesystem watcher delay (probably only availabe in advanced settings), hoping that it’s long enough for the file to actually be written. Or investigate that program and try to get it to update the modification time after write.
1 Like
There’s an interesting sub-aspect of the likely race condition here.
Generally, when a file changes after having been scanned but without Syncthing able to notice, this will be corrected. The receiving side will ask for a block with hash xyz, the sending side will read the block from disk and verify it, seeing a different hash. The block transfer fails, the sending side rehashes the file, upon next retry everything works out.
This doesn’t happen if the block is hashed as all zeroes to begin with though, because we don’t transfer such blocks over the network at all. The receiver just skips writing anything to disk and the block remains all zeroes…
1 Like
Does it mean that we should do some changes concerning the configuration? What are recommended values for ‘rescanIntervalS’ and ‘fsWatcherDelayS’ if we have a lot of small files in a high frequency?
If the problem is about what @imsodin described above, I don’t think you can rely on rescanIntervalS at all. This is because there’s no guarantee that the rescan won’t be triggered by Syncthing while one of your files has just been partially written. You could, in theory, disable periodic scans altogether and rely on fsWatcherDelayS (delayed by e.g. 5 minutes) only. Syncthing will still perform a full rescan on start and/or automatic upgrade though.
you could also disable periodic scans/file watchers and trigger a rescan via REST API once export of XML files is complete
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.