Out of sync items not progressing

[AndiDog (Andreas Sommer)]

I have lots of out-of-sync items shown for transfer to a peer, but none are progressing and the count remains the same.

Device A (macOS / ExFAT):

Device B (Raspberry Pi / Linux / NTFS):

This situation arised after having file permission trouble with the badly-maintained ntfs3 filesystem on Linux where it wouldn’t respect user/group/chmod flags and Syncthing failed to sync a few files (but not all). Therefore, I changed my disk mount back to ntfs-3g where permissions are much easier to handle.

I saved the state of one affected file with curl --fail-with-body -s -X GET -H "X-API-Key: ${apikey}" "http://127.0.0.1:8384/rest/db/file?folder=xxx&file=foobar/whatever.jpg" on each of the devices (a.json and b.json):

$ diff <(jq .local b.json) <(jq .global b.json) && echo "No difference"
No difference


$ diff <(jq .local a.json) <(jq .global a.json) && echo "No difference"
No difference


$ diff -U0 <(jq .global a.json) <(jq .global b.json) && echo "No difference"
--- /dev/fd/63	2024-02-13 12:13:31
+++ /dev/fd/62	2024-02-13 12:13:31
@@ -5 +5 @@
-  "inodeChange": "2009-06-12T17:55:06+02:00",
+  "inodeChange": "2017-10-22T14:00:18.5667277+02:00",
@@ -9 +9 @@
-  "modifiedBy": "MRMPTP5",
+  "modifiedBy": "5FH2SUZ",
@@ -12 +12 @@
-  "noPermissions": true,
+  "noPermissions": false,
@@ -13,0 +14 @@
+  "permissions": "0770",
@@ -22 +23 @@
-  "sequence": 1210,
+  "sequence": 69284,
@@ -26 +27 @@
-    "MRMPTP5:1705429511"
+    "5FH2SUZ:1706256259"


$ diff -U0 <(jq .local a.json) <(jq .local b.json) && echo "No difference"
--- /dev/fd/63	2024-02-13 12:14:06
+++ /dev/fd/62	2024-02-13 12:14:06
@@ -5 +5 @@
-  "inodeChange": "2009-06-12T17:55:06+02:00",
+  "inodeChange": "2017-10-22T14:00:18.5667277+02:00",
@@ -9 +9 @@
-  "modifiedBy": "MRMPTP5",
+  "modifiedBy": "5FH2SUZ",
@@ -12 +12 @@
-  "noPermissions": true,
+  "noPermissions": false,
@@ -13,0 +14 @@
+  "permissions": "0770",
@@ -22 +23 @@
-  "sequence": 1210,
+  "sequence": 69284,
@@ -26 +27 @@
-    "MRMPTP5:1705429511"
+    "5FH2SUZ:1706256259"

Restarting Syncthing didn’t help. Toggling “Ignore file permissions” to force a rescan didn’t help. Nothing in the logs.

touch-ing an affected file makes its modification timestamp get actively synced and therefore reduces the total “out of sync” count by one. Not a great solution because I don’t want to rewrite each file’s timestamp. touch -a (only access timestamp) does nothing – I guess Syncthing only cares about modification timestamp which is reasonable.

On the left-hand side (Folders widget), no file transfers are shown.

Is there a way to reconcile the database/state, since both local and global state is identical on each device? I couldn’t find a REST API endpoint to enforce that. FolderSummary of the Events API seems to contain the out-of-sync items but I have no idea how to change that state. My idea is to bump the files’ sequence number in the Syncthing database, or fix the bug overall by having Syncthing mark files as synced once they’re identical.

[calmh (Jakob Borg)]

Fascinating; either side seems to think they have the latest version and the other doesn’t, and it doesn’t get reconciled since … I don’t know, but apparently the difference is something minor in the metadata. Maybe an effect of enabling ignore-permissions after the fact or something like that…

I saw a similar case the other day, but that revolved around conflict resolution and deleted files, which seems to not be the case here. If you’re fine with sharing it privately I’d be happy to take a look at the zipped databases from each device before we fix it, to figure out what happened.

In any case, I suspect you can resolve this by removing the folder on one side and then re-adding it again, with a restart of that side in between for good measure. That’ll force it to rescan and reevaluate all the files, and then reconcile it around one set of timestamps and whatnot.

[AndiDog (Andreas Sommer)]

I can’t share the database as-is, unfortunately, because it contains lots of private data. Is there something I can check in there? Otherwise, I’m happy to try removing and adding back the folders after taking a backup snapshot.

[AndiDog (Andreas Sommer)]

I’ve removed the folders on both sides, and then re-added them. This worked to re-scan everything from scratch again, using new folder IDs. No more issues right now. Thank you for the idea!