Troubleshoot a few items not syncing

Support
1

I am a new syncthing user and am trying to sync a large folder(590,000 items). I originally synced the folders on the 2 machines with rsync, then fired up syncthing on both machines to keep the folders in sync. On both of the machines the GUI says that the folder is up to date with the global state, but on one of the machines the other machine is marked as syncing 99% with a small number of items outstanding (792 items).

Checking a number of these files on each machine appears that they are in sync with each other, including file permissions and md5sum. Both machines are set to full sync (read and write) and I have nothing set to ignore.

I am not sure how to continue to investigate this. Any suggestions?

2

Further…

Looking at the “out of sync” files list on dirac’s GUI for geo-fun.org shows the “Mod. Device” to be dirac. Looking at the “out of sync” files list for dirac on the geo-fun.org GUI shows mod device as BGRPGB2, which looks like the first part of a device id, but does not match part of dirac’s id.

Any suggestions?

3

It would be best if you provided screenshots.

4

Screenshots

dirac
dirac1796×1328 295 KB
geo
geo1847×1361 306 KB
dirac_oos
dirac_oos1797×1698 400 KB
geo_oos
geo_oos1772×1667 417 KB

5

It seems top device is stuck scanning? Or was this just unfortunate timing?

6

I have been trying to troubleshoot this for the last few days. When it settles it always end up with 26.2MB & 792 files out of sync. I have uninstalled, deleted the .config/syncthing directory on both machines then reinstalled, still the same outcome…

7

What I don’t understand is why the Documents folder is in sync, but the remote machine isn’t (in both directions)

8

Unfortunate timing, here is a later screenshot

dirac2
dirac21814×1352 294 KB

9

‘touch’-ing a file in the “Out of sync items” list causes the file to be synced across both devices. I have just touched one on dirac and the number unsynced has dropped from 792 to 791, but the file doesn’t appear as the “Latest change” in the “Documents” folder despite the access time being updated across both devices.

10

You should query db API for one of the files on both sides to see if there is anything obvious. Also, check the logs.

11

Can you give me an example of the curl command to use? The documentation is a bit sparse. I have tried:

curl -X POST -H "X-API-Key: <API Key>" -d '{"file": "<filename>"}' http://127.0.0.1:8384/rest/db/file/

and I get a 404

12

A simple search goes a long way:

13

So, I think that there is somehow still a reference to potentially an old syncthing instance? On a number of the documents there are references to BGRPGB2, which isn’t one of the current instances. This may not be the cause, but not sure why it is happening.

The documents which are “out of sync” have a global availability of null, which I am guessing is the reason why it cannot sync, but the file is actually available at both machines.

So the next question is…

Why do these files have an availability of null?

this post mentions "availability": null, but says it is expected??

14

I guess posting the same query from both sides might show something interesting.

15

Please post the query results and a list of currently active devices (i.e. the first “block” of their IDs).

16

Active devices

  • DZGK2SA - geo-fun
  • O3ET7QT - dirac
  • 7TUHXYK - vance-X230 (Not sharing the documents folder)
  • BGRPGB2 - Unknown, maybe a previous installation

Example of document that isn’t syncing

From geo-fun 

curl "http://127.0.0.1:8384/rest/db/file?folder=cmpcj-twatt&file=20-29_Interests/21_Art/21.03_Flowers/Daffodil.ai" \
  -H "X-API-Key: xxxxxxxxx"

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2009-03-08T17:23:26Z",
    "modifiedBy": "BGRPGB2",
    "mustRescan": false,
    "name": "20-29_Interests/21_Art/21.03_Flowers/Daffodil.ai",
    "noPermissions": false,
    "numBlocks": 10,
    "permissions": "0644",
    "sequence": 1675496,
    "size": 1300419,
    "type": "FILE",
    "version": [
      "BGRPGB2:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2009-03-08T17:23:26Z",
    "modifiedBy": "BGRPGB2",
    "mustRescan": false,
    "name": "20-29_Interests/21_Art/21.03_Flowers/Daffodil.ai",
    "noPermissions": false,
    "numBlocks": 10,
    "permissions": "0644",
    "sequence": 1675496,
    "size": 1300419,
    "type": "FILE",
    "version": [
      "BGRPGB2:1"
    ]
  }
}

From dirac 

curl "http://127.0.0.1:8384/rest/db/file?folder=cmpcj-twatt&file=20-29_Interests/21_Art/21.03_Flowers/Daffodil.ai" \
  -H "X-API-Key: xxxxxxxx"

 {
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2009-03-08T17:23:26Z",
    "modifiedBy": "O3ET7QT",
    "mustRescan": false,
    "name": "20-29_Interests/21_Art/21.03_Flowers/Daffodil.ai",
    "noPermissions": false,
    "numBlocks": 10,
    "permissions": "0644",
    "sequence": 632068,
    "size": 1300419,
    "type": "FILE",
    "version": [
      "O3ET7QT:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2009-03-08T17:23:26Z",
    "modifiedBy": "O3ET7QT",
    "mustRescan": false,
    "name": "20-29_Interests/21_Art/21.03_Flowers/Daffodil.ai",
    "noPermissions": false,
    "numBlocks": 10,
    "permissions": "0644",
    "sequence": 632068,
    "size": 1300419,
    "type": "FILE",
    "version": [
      "O3ET7QT:1"
    ]
  }
}

Analysis

It looks like DZGK2SA (geo-fun) thinks the latest version belongs to BGRPGB2 (Old install). O3ET7QT (dirac) thinks that it has the latest version. Maybe this conflict in state is causing availability: null for the file?

I am not sure how geo-fun has reference to the old install. I thought I had cleared out all of the state on both machines whilst syncthing was shutdown, then restarted both. The other machine isn’t an introducer, so the old reference should not have propagated that way.

Can you suggest how I can resolve this?

17

The weird thing here is that they disagree on the version vector of the global file. Whether or not the key in the version vector is an old device isn’t really relevant (it might be a pointer to how we got here though). What should happen is that one of these file infos “wins” (despite being equal, there is always a winner) and that triggers a conflict, which is then silently resolved. However here either they didn’t send the info or there was a problem “accounting” that. This comes up way too often, but still rare enough that I haven’t noticed any pattern…

Can you start Syncthing on both sides with -reset-deltas please. If it comes up again we can start debugging :smiley:

18

The modification timestamp on the two are identical, which is the deciding factor when picking who wins the conflict. I wonder if the logic there when the timestamps are the same is somehow flawed.

19

Or it just simple assumes the file is not changed (even if the version vectora are different), because size mtime et al matches.

20

Starting both with -reset-deltas has made no difference, still 791 files out of sync.

log files on both sides shows:

[O3ET7] 16:12:30 INFO: Reinitializing delta index IDs

but when the scanning settles we are back at the same point.

Regarding @AudriusButkevicius comments there must be something a little random or a race condition otherwise the whole filesystem would be in this state. As I said in the earlier post I rsync -a the files across before starting syncthing, so the timestamps and file permissions should be identical on both sides.