Ingored files stay "out of sync"

Just so I get the nomenclature right:

Win Laptop 1 (L1) shows the workstation (W) as syncing. Both L1 and W show the folder as up-to-date. And W shows L1 as up-to-date. Any mistakes?

It seems like the ignore lists are equal from the screenshots, are they? (Asking because of the vastly differing local state numbers)

Can you query the /rest/db/file endpoint for one of the out-of-sync files on both on L1 and W? https://docs.syncthing.net/dev/rest.html

The sync has been in this “syncing”-state for weeks without any progress. Several restarts/reboots have occurred, without help. The ignore lists on LW/W1 are almost equal, but not the same. The both ends (L 1, W 1) have the files in “out-of-sync” state in their ignore list.

I do the DB reset next (on L 1). Btw. Until the sync handling is super-robust, it would be nice to have the DB reset as a feature e.g. under advanced settings.

Before you do a db reset, do an index reset (-reset-deltas). That’s much less resource intensive and might already clear things up.

As the folder on the other side says up to date, there is no actual syncing and thus no progress to be exptected - this is an incorrect state. And the information from /rest/db/file might give some clues as to what is wrong. However you can also just do the index reset and hopefully that will clear things up.

Regardless, it would be interesting to see the output as it smells like a bug.

What “output” should I send?

An intermediate report that doing -reset-deltas to all computers did not help.

It seems you have two folders, are both folders shared with both devices?

The output of the /rest/db/file API endpoint for one of the ignored files, from both sides, would be interesting.

I am strugging with the REST API:

curl -X POST -H “X-API-Key: KEYISHERE” ‘http://127.0.0.1:8385/rest/db/file?folder=FOLDER_ID&file=2%20WORK

Returns

404 page not found

Please note the syncthing server is running on port 8385 since I have two instances running under different users. API-KEY and folder (ID) have been masked.

I think its a get not a post

1 Like

Well. L1 says a file X is “out of sync” under the "Remote devices panel on Server 2. When I run the GET rest/db/file on that file (ULRencoded filename):

MOBILE SYNC\2019\07\20190712_075126444_iOS.jpg ==> MOBILE%20SYNC%5C2019%5C07%5C20190712_075126444_iOS.jpg

I get No such object in the index on Server 2 and L2 replies temporary redirect.

Use forward slashes for path separators. Regarding the redirect: Do you use https? Then you need the -k flag.

I managed the get the GET rest/db/file request to work on the Server 2 (Linux), but not on Laptop 2 (Ubuntu shell in Win10). The Laptop 2 keeps saying temporary redirect. I run the same command over remote SSH tunnel from Server #2 (Linux) and I got the same “temporary redirect” reply so therefore it must be a “legit” response, not a shell quotation issue.

Server #2 says:

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": true,
    "localFlags": 0,
    "modified": "2019-07-12T10:51:26+03:00",
    "modifiedBy": "WIN WORKSTATION",
    "mustRescan": false,
    "name": "MOBILE SYNC/2019/07/20190712_075126444_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 0,
    "permissions": "0640",
    "sequence": 300193,
    "size": 2903726,
    "type": 0,
    "version": [
      "SERVER #1:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "1970-01-01T02:00:00+02:00",
    "modifiedBy": "",
    "mustRescan": false,
    "name": "",
    "noPermissions": false,
    "numBlocks": 0,
    "permissions": "0",
    "sequence": 0,
    "size": 0,
    "type": 0,
    "version": []
  }
}

And sorry for not knowing how to quote a block nicely

And sorry for sneakily correcting the formatting. Tripple backticks ``` before and after do the trick (or the preformatted text thingy in the “editor”).

Did you check the https thing (-k flag)? Otherwise on windows you can do it in power shell like this: Again, The remote device keeps displaying the synchronization(95%) It would be very helpfully, I have an idea about what it might be about (globally invalid files not being needed), but not yet what exactly is wrong - getting the info from the other side will hopefully help.

1 Like

Sorry for being slow here. The Windows machines are using SyncTrazor and I was unaware those Syncthing GUI servers actually are behind HTTPS. Finally got it working.

Here is GET rest/db/file for the very same file from Laptop #1 (the other end):

{
  "availability": [
    {
      "id": "SERVER #1",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2019-07-12T10:51:26+03:00",
    "modifiedBy": "SERVER #1",
    "mustRescan": false,
    "name": "MOBILE SYNC\\2019\\07\\20190712_075126444_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 23,
    "permissions": "0640",
    "sequence": 513241,
    "size": 2903726,
    "type": 0,
    "version": [
      "SERVER #1:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": true,
    "invalid": true,
    "localFlags": 2,
    "modified": "2019-07-12T10:51:26+03:00",
    "modifiedBy": "LAPTOP #1",
    "mustRescan": false,
    "name": "MOBILE SYNC\\2019\\07\\20190712_075126444_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 0,
    "permissions": "0640",
    "sequence": 274445,
    "size": 2903726,
    "type": 0,
    "version": [
      "SERVER #1:1"
    ]
  }
}

So if I read these correctly both the Server #2 and Laptop #1 think there is even newer version of the file modified by WIN WORKSTATION that has the file under Ignored Files. I kind of think that remote computers should consider file modification by computers having the file on Ignored Files list, but this must be more complicated than that.

I have now set up a SSH tunnel between Servers 1/2 so those can finally sync each other. That may sort out my problem although I think Syncthing does not work here as it should.

Thanks!

Are you sure you didn’t make any mistakes when obscuring the IDs? Ideally just leave the first block of the ID, that’s not sensitive and makes mistakes virtually impossible. Reason I am asking is:

Laptop 1 has server 1 as an availability, while Server 1 shows the file as globally invalid and doesn’t have it at all locally - those two things should be mutually exclusive. Also “WIN WORKSTATION” is coming up, but never it’s ID.

Yes, I made a mistake:

Server #2 view:

{
  "availability": [
    {
      "id": "WJXMAFI-....",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2019-07-12T10:51:26+03:00",
    "modifiedBy": "WJXMAFI",
    "mustRescan": false,
    "name": "MOBILE SYNC/2019/07/20190712_075126444_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 23,
    "permissions": "0640",
    "sequence": 513241,
    "size": 2903726,
    "type": 0,
    "version": [
      "WJXMAFI:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "1970-01-01T02:00:00+02:00",
    "modifiedBy": "",
    "mustRescan": false,
    "name": "",
    "noPermissions": false,
    "numBlocks": 0,
    "permissions": "0",
    "sequence": 0,
    "size": 0,
    "type": 0,
    "version": []
  }
}

Laptop #1’s view:

{
  "availability": [
    {
      "id": "WJXMAFI-...",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2019-07-12T10:51:26+03:00",
    "modifiedBy": "WJXMAFI",
    "mustRescan": false,
    "name": "MOBILE SYNC\\2019\\07\\20190712_075126444_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 23,
    "permissions": "0640",
    "sequence": 513241,
    "size": 2903726,
    "type": 0,
    "version": [
      "WJXMAFI:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": true,
    "invalid": true,
    "localFlags": 2,
    "modified": "2019-07-12T10:51:26+03:00",
    "modifiedBy": "WGNJRUK",
    "mustRescan": false,
    "name": "MOBILE SYNC\\2019\\07\\20190712_075126444_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 0,
    "permissions": "0640",
    "sequence": 274445,
    "size": 2903726,
    "type": 0,
    "version": [
      "WJXMAFI:1"
    ]
  }
}

IDs

WJXMAFI = Server #1

WGNJRUK = Laptop #1

So Laptop #1 thinks it has the file (it does not) and the file is on ignored list. I am guessing that the ignore list is preventing Laptop #1 to update the state of the file (it does not have it), but yet it believes it has the newest version but since it is on ignore list so it just keeps thinking so, without really doing anything to achieve a steady state.

I had in parallel similar problem with between Laptop #1 and Server #2, and between Workstation and Server #1.

I finally solved the first problem by connecting Server #1 and #2 through a SSH tunnel and letting them sync. The second problem I solved by “un-ignoring” the problematic folder on Workstation letting it sync. After all the syncs were finished I ignored the problematic folder again.

There was also a persistent “failed items” issue at Server #1, but all those files were deleted in all the computers already. For reasons unknown to me, Syncthing did not clear out this situation on its own (over several hours) so I sorted out that (unrelated) issue by forcing resync.

Now everything is in sync, but I still think the out-of-sync items never competing was caused by a bug in Syncthing code. I will continue to keep the servers connected with help of an autossh tunnel (ran as systemd service).

Quite a lot of work :sweat:

Here are GET rest/system/file results from the 2nd (now solved) problem mentioned above between Server #1 and Workstation. These computers are in the same LAN.

Server #1 view:

{
  "availability": [
    {
      "id": "2PSJLZW-....",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2017-08-01T13:29:11+03:00",
    "modifiedBy": "WJXMAFI",
    "mustRescan": false,
    "name": "MOBILE SYNC/2017/08/20170801_102911019_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 6,
    "permissions": "0640",
    "sequence": 150889,
    "size": 775253,
    "type": 0,
    "version": [
      "WJXMAFI:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2017-08-01T13:29:11+03:00",
    "modifiedBy": "WJXMAFI",
    "mustRescan": false,
    "name": "MOBILE SYNC/2017/08/20170801_102911019_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 6,
    "permissions": "0640",
    "sequence": 413679,
    "size": 775253,
    "type": 0,
    "version": [
      "WJXMAFI:1"
    ]
  }
}

Workstation’s view:

{
  "availability": [
    {
      "id": "2PSJLZW-...",
      "fromTemporary": false
    },
    {
      "id": "WJXMAFI-...",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2017-08-01T13:29:11+03:00",
    "modifiedBy": "WJXMAFI",
    "mustRescan": false,
    "name": "MOBILE SYNC\\2017\\08\\20170801_102911019_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 6,
    "permissions": "0640",
    "sequence": 150889,
    "size": 775253,
    "type": 0,
    "version": [
      "WJXMAFI:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": true,
    "invalid": true,
    "localFlags": 2,
    "modified": "2017-08-01T13:29:11+03:00",
    "modifiedBy": "6T76ERY",
    "mustRescan": false,
    "name": "MOBILE SYNC\\2017\\08\\20170801_102911019_iOS.jpg",
    "noPermissions": true,
    "numBlocks": 0,
    "permissions": "0640",
    "sequence": 201912,
    "size": 775253,
    "type": 0,
    "version": [
      "WJXMAFI:1"
    ]
  }
}

IDs

WJXMAFI = Server #1

6T76ERY = Workstation

2PSJLZW = Server #2

The latest db/file outputs look fine: Everyone is on the same version, laptop having it ignored, the servers not.

The one before (Ingored files stay "out of sync") does show a problem: Both sides have the same, valid file, but only laptop has a local file (ignored). The server should also have a local file, even if ignored, but doesn’t. I assume you did restarts or pause/unpause the folder (that force triggers a sync). Now the difficult thing is to find out why it doesn’t sync/ignore it and one step towards finding out would be logs with model debug facility enabled during a sync operation. However as I understand you have fixed the problem, so it no longer exists anywhere?

1st problem

Hi, unfortunately by fixing the error state I also destroyed an opportunity to debug this deeper.

The first problem seemed to be because the Server #2 wanted the files that only Server #1 had, but had not connection then (now they have). Laptop #1 used to have the files, but I have since ignored those on the laptop.

I also think I have misunderstood what the “progress” means. I have though it means the status of point-to-point sync, but it may mean a general sync situation of the other end. IF this is the case, I do not undertand why all the computers do not show the same sync progress for a particular computer.

2nd problem

What I don’t understand why the 2nd problem between Server #1 and Workstation did not clear out since those are in the same LAN. The problem had persisted for weeks.

But all is good now. I think I managed to get those problems when ignoring folders without having all the computers in sync. Next time I try to be patient to enable proper debugging. Thanks for all the patience.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.