I have a shared folder (for the Obsidian notes app) across all my three devices (Windows PC, Android phone and Android tablet). I get sync-conflicts all the time, even when I’m sure that I edited the file on a single device only. Do I use a wrong synchronization strategy?
On each of the devices, I share the Obsidian folder with the other two devices. Cannot this be the cause? Shouldn’t I use a “star” sync strategy, i.e. not to synchronize the Androids directly with each other, but through the central PC only? Would this help?
My theory is this may be caused by the fact that the devices mutually synchronize in different times. Say I have devices A, B and C.
I make a change on A, and that change propagates to B.
I make another change on A, and that propagates to C.
Now, when B and C try to synchronize, there is a conflict.
I don’t know whether Syncthing is able to handle this type of conflicts.
However, I would like the Androids to synchronize even when the PC is off.
This shouldn’t matter at all, and normally it’s also more efficient to connect all devices together.
This shouldn’t lead to a conflict unless both B and C have modified the file before trying to sync it with each other.
Just to confirm, are you running the newest version of Syncthing everywhere? Have you tried to enable the “Ignore Permissions” option in the Advanced tab in the folder settings on all devices?
On Android devices, I have the latest Google Play version of Syncthing. On Win10, I have SyncTrayzor with Synthing 1.27.3.
As I synchronize Win with Android, I have “Ignore Permissions” set on Android. I didn’t have it set on Win though, so I turned it on. I’ll let you know whether there’s a difference.
This is the folder status from my PC, where I was editing the Obsidian.md file.
As said before, this folder is synced with my tablet and my phone (and they are mutually synchronized too), but I edited the file on my PC only.
These are the only three messages in the Synthing log around 12:30:
[YYYTN] 2024/02/18 12:30:59 INFO: Puller (folder "Obsidian" (myb9l-xxxxx), item "Notes\\Obsidian\\Obsidian.md"): syncing: finishing: checking existing file: file modified but not rescanned; will try again later
[YYYTN] 2024/02/18 12:30:59 INFO: "Obsidian" (myb9l-xxxxx): Failed to sync 1 items
[YYYTN] 2024/02/18 12:30:59 INFO: Folder "Obsidian" (myb9l-xxxxx) isn't making sync progress - retrying in 1m0s.
It still seems to me that the triangle synchronization is the cause. I’ll try to switch off the sync between the Android devices.
Edit: The ~syncthing*.tmp file above is called .syncthing*.tmp file on the tablet, and has a later timestamp 12:36.
The strange thing is that it is full of zeroes (\x0), as well as the second sync-conflict file. I remember that under some circumstances, the original file was completely empty, the only content left was in the sync-conflict file.
I think this may not help at all. The reason is that even if you don’t connect the Android devices with one another, their changes are still going to sync as before, the only difference being that they are going to transmit through the PC.
For testing purposes, I would suggest trying to disconnect one of the Android devices from Syncthing completely, and then continue using the application only on the PC and the other Android device first. This way, you will be able to verify whether there are still conflicts when using just the two devices.
Actually, according to the logs, the phone didn’t participate at all, as it has no log record between 11:30 and 15:20. Thus the problem emerged between the PC and tablet only. I’ll try switching on and off various settings, but It shall take some time.
What is the .synthing.*.tmp file? Is it the file where Syncthing stores the partial content before it gets completely synchronized?
It starts to be a pretty serious problem. Now Syncthing deleted one of my files on the tablet, probably while synchronizing it from the PC.
I have edited the file on the PC, and the tablet started showing Out-of-sync status of the folder. The tablet Web GUI stated that it cannot replace a non-existing file. Indeed, the original file was deleted, but there was left the corresponding .synthing.*.tmp file with the changes coming from the PC.
I’m not aware of having set any non-standard settings.
I’ll keep tracking down the cause, but now I’m really worried.
By investigating the issue I identified many situations where the Syncthing synchronization fails, without any warning. The reasons are various, basically in two cathegories (maybe more):
Let’s have: notebook (N), tablet (T) and mobile phone (M).
Syncthing competes with another “aggressive” application. (The application is the Folder notes plugin for Obsidian.)
1a. Whenever the application sees an empty folder, it creates a default file in it.
The problematic sync scenario is as follows:
i. At N, lets create a new folder with the default file in it, Xxx/Xxx.md.
ii. Syncthing (ST) replicates the structure to M.
iii. ST first creates the folder Xxx/
iv. Now the Obsidian plugin steps in, it sees an empty folder, so it creates a new empty default file in it, Xxx/Xxx.md.
v. ST wants to replicate the Xxx/Xxx.md, but it is already there, and a newer one. Hence a sync-conflict file is created.
1b. There are other similar problems that seem to be caused by the following scenario:
i. While replicating a file Zzz.md from N to M, ST creates the syncthing.Zzz.md.tmp file.
ii. When the transfer is finished, ST seems to delete the old version of Zzz.md at M in order to replace it with the new version.
iii. At this moment, the Obsidian plugin steps in, and recreates the deleted file, with a new timestamp.
iv. ST is unable to replace the old file, thus a sync-conflict occurs.
1c. Similar to 1b. When ST wants to replace an obsidian configuration file, Obsidian detects that, recreates the file and causes sync-conflict.
The worst case in 1b and 1c are, when the file is recreated empty, or with the same lenght but filled with zeroes (\x0).
The partial solution for 1a. and 1b. is to turn off the aggressive file creation in the Folder notes plugin. The feature is actually handy.
For 1c, I have no solution, the aggressive recovery of the configuration files cannot be turned off. I can make a ST exception for some of the config files, but this is only a partial solution.
The second cathegory of sync problems is caused by sharing folders inside already shared folders (and has no relation to Obsidian).
I have the Obsidian folder shared N ↔ T ↔ M ↔ N. But I also have its superfolder Documents shared M ↔ T, and this is the cause.
In Sync a subfolder of an already synced folder - #3 by Alex, there’s a warning, but not a serious one. In fact it turns out, that in this case synchronization can cause a lot of conflicts and even more serious troubles. For instance:
2a. Files and even folders can disappear.
Scenario:
i. I create a directory tree at T
ii. ST replicates it to N
iii. At N I validate it, if it is not good enough, I delete it.
iv. The deletion is replicated back to T.
v. At T I recreate a new version of the directory tree.
vi. Now, the synchronization of the superfolder steps in, and replicates deletion of the directory from M.
The more serious situation may be when the M is disconnected for some time, and when it connects, it deletes the final version of the directory tree.
To prevent this, the Obsidian subfolder must be ignored in the synchronization of the Documents superfolder.
Conclusion
I think, there should be a more serious warning written somewhere, e.g. in FAQ - don’t do this, if you do not want to lose data.
There are also other problems that I haven’t analyzed yet to learn the cause.
Seems like what you are trying to do just won’t work
All of the issues, including the nested folder sync issue, happen because you have two automated things stepping on each other toes, not because either of them are inherently wrong on their own.
Filesystems are not transactional on anything else than a file level, hence I don’t think this can be fixed in anyway.
I agree, what I describe in case 1b. is caused by the fact that the file operations are not atomic. But I haven’t chosen the way how Obsidian works.
In case 2., the warnings in FAQ and in the forum weren’t strong enough.
I have written all this here, in order to be documented somewhere. It would be worth extending the note in FAQ with the info that sync-conflicts and data loss may occur, and that it has been already proven. If I read that, I would avoid that.
