Yesterday before going to sleep I decided to re-hash my syncthing folders.
I use this to detect bit rot and to make sure some super large files (containers) are bit-by-bit correctly stored.
Usually i never have any sync conflicts after doing this.
But I since all my syncthing devices are on .48 I decided to enable large blocks on the device I was re-hashing everything throughout the night.
After waking up, like all files larger than 300 MB and those up to 30 GB were copied to my corresponding server with sync-conflict in it.
Only one folder on my server didn’t have one conflict - the one and single folder I enabled Large Blocks some weeks ago when I let the machine re-hash everything.
It look for me like there is a problem with Large Blocks if the option is disabled on one side.
I checked the files on my both PCs with Total Commander (bit-by-bit). All are absolutely the same. Either checked between the sync conflicts or the original files.
That means that there is some kind of hash check mistake? Or the other side does not correctly use the large blocks since it doesn’t have hashed large blocks yet?
Do I have to enable large blocks on all devices and let everything rehash again?
There is no warning that the other side doesnt have large blocks yet or has to rescan itself the files because the starting side sends large blocks only?!
Im confused :).
So… First of all, if you enable this you should do so on both sides, at roughly the same time. Otherwise things will be very suboptimal.
That said I think there is a different underlying problem here, with permissions or modification times on one of the sides, that simply hasn’t showed up before now.
Sync conflicts happen when a file has changed on both sides between syncs. That change can be in contents, or just metadata (e.g. a single permission bit). Upon detecting the sync conflict Syncthing will create a sync conflict copy, unless the contents are anyway the same. I think this is probably what’s been going on in the background already. However, when you change the block size on one side, the contents cannot be the same (block wise), so suddenly what was a harmless metadata conflict becomes a real content conflict.
The documentation doesn’t state that you have to enable it on both sides, per folder, at the same time :(.
Now I understand that if one side re-hashes everything with large blocks, the other side cannot use the large blocks for checking if everything is in sync. It doesn’t rescan the files because for it large blocks are disabled (?).
In my case, only files bigger than 300 MB are doubled and have a sync-conflict. I know about permissions. Both sides are Windows PCs and modification times i have checked with Total Commander. I use robocopy /mir too. I don’t think there is a time problem.
Ok, so what has happened:
PC1 re-hashes with large blocks. PC2 doesn’t know what he receives.
PC2 pulles the entire file and adds sync-conflict to the name.
Conclusion: Re-hash everything with large blocks enabled everywhere even on my Android phone ^^.
Well, ok :D.
Sync file >256MiB between two instances with large blocks disabled.
On one instance, stop it, remove the db, enable large block sizes and start again.
Result: Sync conflict.
Actually in hindsight isn’t it obvious?: On one-sided db recreation every file is in conflict (regardless of large blocks). Without large blocks that wouldn’t be an issue, but now with one-sided large blocks, the content of all files will look different -> conflict copy is created.
Yeah exactly.
It was before going to sleep and I just thought, when I now delete the DB and let everything re-hash over night, why not actually use the large blocks now?
It didn’t come to my mind to enable large blocks on my other PC or even have to rescan there everything too.
The thing that made this problem appear for you and not everyone else, is that you remove your db. That’s not required to enable large blocks, so probably nobody else did it together.
I believe the only clean way to solve this is to make large blocks a folder cluster wide setting (i.e. enabling only really takes effect once everyone has it enabled), but that seems too complicated for the gain (and it’s anyway too late).
Bottom line: Don’t reset the database and enable large blocks at the same time.
All of this works as expected to me. The file content differs as the blocks don’t match, hence conflicts. The user is to blame here for doing something adventurous.
Nope, blaming the user is definitely wrong. I agree on the consequence: This is too niche a scenario to care about. However it was a design choice to allow sharing between folders with and without large blocks enabled, with the reasoning that all that can happen is regular full rehashs. Now it is apparent, that there is another scenario, in which versioning conflicts with identical file contents lead to conflict copies. Sure, from an internal perspective it’s perfectly clear that this will happen, but not from the user perspective.
In essence I am not arguing about the technical decision, I am just arguing about communication. It should be something along the lines of “Yeah, you found an edge case where things go south. Please just don’t trigger that edge case anymore so we can consider it settled - thanks.”
But there is nothing to fix here and no form of prevention other than breaking compatability between versions that do and do not support large blocks.
This only happened because the user decided to change the representation of files AND drop the database. The same would have happened if he chmod -r’ed or mass touched the files and reset the database.
Yes, resetting the database can lead to conflicts, under more than one scenario, but this is not business as usual activity he should be doing, and thats what caused it.
So I missed that (implied?) detail - that the database was deleted. This is pretty much never a good idea and will indeed always cause sync conflicts. Syncthing is nice enough to handle those conflicts automatically when the content doesn’t differ, but not when the block size is changed.
Without blaming anyone I still think this is as intended.