syncthing sync identical md5

cedricbrx · October 20, 2019, 4:17pm

I have 2 servers, the two are Debian 10. The second i installed yesterday is with installed with snapraid. I synced the content of the first server to the second. Everything went well. Snapraid told me about zero sub-second timestamp (https://www.snapraid.it/faq#subsecond) and touched some files. After that a lot file were synced, despite they have the same md5 and the creation-time are the same. Souldn’t only the metadata be synced?

xx@mars:~/Programs$ stat “./.stversions/MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08~20191020-162127.rar” File: ./.stversions/MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08~20191020-162127.rar Size: 525336576 Blocks: 1026048 IO Block: 4096 regular file Device: fd00h/64768d Inode: 129237020 Links: 1 Access: (0644/-rw-r–r--) Uid: ( 1000/ xx) Gid: ( 1000/ xx) Access: 2019-10-20 16:52:39.985386455 +0200 Modify: 2019-03-05 18:35:40.000000000 +0100 Change: 2019-10-20 16:21:27.056549349 +0200

xx@mars:~/Programs$ stat “./MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08.rar” File: ./MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08.rar Size: 525336576 Blocks: 1026048 IO Block: 4096 regular file Device: fd00h/64768d Inode: 129239964 Links: 1 Access: (0644/-rw-r–r--) Uid: ( 1000/ xx) Gid: ( 1000/ xx) Access: 2019-10-20 16:49:49.966758381 +0200 Modify: 2019-03-05 18:35:40.528936260 +0100 Change: 2019-10-20 16:21:27.056549349 +0200

xx@mars:~/Programs$ md5sum “./.stversions/MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08~20191020-162127.rar” 067ad6ed2ba4722f834f562870dddf0d ./.stversions/MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08~20191020-162127.rar xx@mars:~/Programs$ md5sum “./MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08.rar” 067ad6ed2ba4722f834f562870dddf0d ./MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08.rar

wweich · October 20, 2019, 4:30pm

Does Syncthing actually transfer file data? The “syncing” status can also mean, that Syncthing doesn’t know if they are identical (as it is the first sync) and is only comparing the hashes. Depending on file amount and size, this can take a while, especially if you shared the folder before the initial scan completed.

AudriusButkevicius · October 20, 2019, 5:41pm

Essentially, syncthing syncs more than just content, it also syncs metadata, which is mtime, permissions etc. So even if the content is the same, it can sync the file if the metadata is different.

cedricbrx · October 20, 2019, 7:12pm

There is an option to ignore permission, would it be a possibility to add an option to ignore timestamps if the content is identical.

AudriusButkevicius · October 20, 2019, 7:24pm

No, because timestamp is how we check if the file has changed.

It doesn’t transfer content if only the timestamps differ, so I am not sure why you care about this?

cedricbrx · October 20, 2019, 8:20pm

If you have file versioning enabled, syncthing create a lot of files, which are identical to the previous version. It takes a lot of space and a lot of time to delete the unneeded files

AudriusButkevicius · October 20, 2019, 8:34pm

Syncthing does not version files if only metadata is different I believe.

cedricbrx · October 20, 2019, 8:38pm

i think it does, i have a lot of files in my .stversions folder which are identical (md5sum, creation time) i don’t if my first post was good enought to explain my issue

AudriusButkevicius · October 20, 2019, 8:58pm

github.com

syncthing/syncthing/blob/master/lib/model/folder_sendrecv.go#L376


				}
			}
			changed++


		case file.Type == protocol.FileInfoTypeFile:
			curFile, hasCurFile := f.fset.Get(protocol.LocalDeviceID, file.Name)
			if _, need := blockDiff(curFile.Blocks, file.Blocks); hasCurFile && len(need) == 0 {
				// We are supposed to copy the entire file, and then fetch nothing. We
				// are only updating metadata, so we don't actually *need* to make the
				// copy.
				f.shortcutFile(file, curFile, dbUpdateChan)
			} else {
				// Queue files for processing after directories and symlinks.
				f.queue.Push(file.Name, file.Size, file.ModTime())
			}


		case runtime.GOOS == "windows" && file.IsSymlink():
			f.resetPullError(file.Name)
			file.SetUnsupported(f.shortID)
			l.Debugln(f, "Invalidating symlink (unsupported)", file.Name)
			dbUpdateChan <- dbUpdateJob{file, dbUpdateInvalidate}

disagrees with you, if you folllow the function, you’ll see that it just chmods and chtimes.

AudriusButkevicius · October 20, 2019, 9:06pm

is the file perhaps flip flopping between two similar hashes?

cedricbrx · October 20, 2019, 9:09pm

it has the same hash md5sum “./.stversions/MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08~20191020-162127.rar” 067ad6ed2ba4722f834f562870dddf0d ./.stversions/MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08~20191020-162127.rar

md5sum “./MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08.rar” 067ad6ed2ba4722f834f562870dddf0d ./MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08.rar

The only difference is the modification time: File: ./.stversions/MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08~20191020-162127.rar Modify: 2019-03-05 18:35:40.000000000

File: ./MB WIS ASRA 11 2018-4DDL/MB WIS ASRA 11 2018-4DDL.part08.rar Modify: 2019-03-05 18:35:40.528936260

This change was caused by a touch command due to snapraid as i described

AudriusButkevicius · October 20, 2019, 9:19pm

If you have repro steps that I could run, I’d be interested at looking into this.

cedricbrx · October 20, 2019, 9:22pm

Ok, i will do that tomorrow, do you need any logs?

AudriusButkevicius · October 20, 2019, 9:26pm

Not really, as long as I have detailed step by step guide how to reproduce it on a new isolated folder with simple steps.

cedricbrx · October 21, 2019, 9:25am

I tested on 2 debian machines with newest syncthing installed and you were right. The touch command doesn’t make synchting have files in the .stversions folder. The only which changes is that i’m running a syncthing instance on a synology NAS and the changes came from that instance. Perhaps DSM is doing something wrong, but that’s not your business.

system · November 20, 2019, 9:25am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.