How does conflict resolution work?

General
1

Hey Syncthing Team,

I’m a long-time Syncthing user and fan. I’m trying to deepen my understanding of the system so I can rely on it for more of my data, and I’ve been wondering about how sync works in different situations.

Syncthing’s FAQ answers “What if there is a conflict?”:

Syncthing does recognize conflicts. When a file has been modified on two devices simultaneously and the content actually differs, one of the files will be renamed to <filename>.sync-conflict-<date>-<time>-<modifiedBy>.<ext> . The file with the older modification time will be marked as the conflicting file and thus be renamed. If the modification times are equal, the file originating from the device which has the larger value of the first 63 bits for his device ID will be marked as the conflicting file. If the conflict is between a modification and a deletion of the file, the modified file always wins and is resurrected without renaming on the device where it was deleted.

Beware that the <filename>.sync-conflict-<date>-<time>-<modifiedBy>.<ext> files are treated as normal files after they are created, so they are propagated between devices. We do this because the conflict is detected and resolved on one device, creating the sync-conflict file, but it’s just as much of a conflict everywhere else and we don’t know which of the conflicting files is the “best” from the user point of view.

Dropbox recently rewrote its sync engine and improved resolution of conflicts, discussed in their blog post. A Dropbox employee commented in 2017 that their client handles “hundreds of special cases”.

I have a lot of questions about how Syncthing’s sync works. What happens when directory trees are rearranged in conflicting ways? What about file and directory renames, symlinks, hardlinks, reflinks, filesystems that support different sets of features, unicode, case sensitivity, bind mounts, divergent metadata?

Maybe if I thought about it long enough I’d see that the FAQ answer resolves all of these questions, but I can’t yet reason about what will happen. In general, how can I predict what will happen to my data under a certain set of changes? Is there an overarching theory-of-syncthing sync that will reveal all this? Where is the relevant code handling these cases?

2

The overarching theory is: Look for the smallest common denominator. E.g. everything is a file (except for directories, but they are anyway mostly metadata, and lets not talk about symlinks) and what Go’s filesystem library does, is what we do (except sometimes). Syncthing generally doesn’t try to represent differing characteristics, it pretends everything is the same. And when there’s stuff that doesn’t behave the same everywhere, Syncthing bends over backwards (very elegantly, mostly) to hide that (e.g. unicode normalization working differently on macs). Case sensitivity is one of the places where Syncthing just knowingly hits the wall head-first repeatedly: It just pretends windows is case sensitive as everything else and that doesn’t work out well (not because that’s fun, but because it’s a nastier problem than it might seem at first glance). The FAQ has an entry detailing what exactly is synced, but essentially it’s data, modtimes und posix permissions unless ignored.

When it comes to renames: That’s also not tracked. There’s stunts on both sides (where the change happens originally by the user and where Syncthing does the change) to make renames detectable/detect renames in a stream of individual file changes, but devices never tell each other that something has been moved/renamed.

As to conflict resolution: That’s mostly just enumeration. Any change gets counted on a device specific counter. If a number increases on two counters, two devices changed the file “at the same time” (before getting in sync), so that’s a conflict. Who wins is determined by “deletion” (file change vs deletion means file change always wins), by modtime or dice roll if equal.

3

As to conflict resolution: That’s mostly just enumeration. Any change gets counted on a device specific counter. If a number increases on two counters, two devices changed the file “at the same time” (before getting in sync), so that’s a conflict. Who wins is determined by “deletion” (file change vs deletion means file change always wins), by modtime or dice roll if equal.

I’m curious exactly how this works when a file is changed by two different devices “at the same time” as described here. If device A edits line 1 of the file and device B edits line 2, such that their actual edits are not in conflict, does Syncthing still treat this as a conflict and create a new sync-conflict file?

I’ve been using Syncthing for over a year and always seemingly get sync conflict files that are nearly identical and need not exist (maybe whitespace is different) , so I’m trying to determine how to minimize this.

4

Syncthing is not content aware. There is no difference between whitespace or some other bytes, it cares about the checksum of the file. A conflict is simply “I have a change locally that you haven’t observed, you have a different change locally that I haven’t observed”, its a conflict. This doesn’t even need to happen at the same time. Shutdown/pause/disconnect one syncthing instance, modify file on both sides, reconnect, you’ll end up with a conflict.

You can minimize conflicts by making sure the two instances are always connected, or by genuinely not modifying files on both sides.

5

I enthusiastically adopted syncthing to replace the synchronization solution of the original Chinese service provider Nut Cloud (jianguoyun.com). Because syncthing solves many problems that Nut Cloud cannot solve, but syncthing also has an annoying problem, that is, it generates a large number of sync-conflicts. When using Nut Cloud, this situation is very rare. How did they do it? of?

“You can minimize conflicts by making sure the two instances are always connected, or by genuinely not modifying files on both sides.”

Both of these are unrealistic. I have multiple instances, at least 5. In most cases, at least 1 or 2 are offline. The instances are online at the same time. When editing the same file, there will be almost no conflicts. , this is great,

But when an offline instance suddenly comes online, a large number of conflicts occur. Even if the target file is not modified in other terminals, it takes ten or twenty minutes a day to delete it and may need to compare it. This is very distressing.

However, given the p2p function of syncthing and its combination with markor, it can realize functions such as browsing markdown images on the Android side, and it is difficult to give up syncthing. I really hope that the development team can consider this issue.

Suggestion: According to my observation, if file1 is modified on instance a, synchronize to instance b, instance a goes offline, instance c comes online, instance c synchronizes from instance b, gets the latest file1, modifies file1, synchronizes to instance b, instance When c goes offline and instance a comes online, the conflict file of file1 is generated, but its conflict file is unnecessary in my opinion. Maybe it is necessary in some scenarios. I suggest adding an option to not retain conflict files, which is left to the user’s choice. Under normal circumstances, the historical version retention function has already saved conflict files. I will choose not to retain conflict files and enable historical versions.

I saw that there is an option of maxConflicts. I can try it.

1 Like