"Out Of Sync" - First use of Syncthing beween Synology and W10

Hi,

Context: I’m looking for a sync solution between several PCs and my NAS, and selected Syncthing for its features and openness, so this is my first try, but I cannot seem to make sync work…

What I did: Installed v1.5 community package on Synology 1812+, and SyncTrayzor 1.1.24 (Syncthing v1.5) on W10 64b. I’ve added each device on the other one, and set up a single sync folder called ‘Audio’ on both. That folder had been previously sync’ed on demand using a manual tool, so contents was mostly identical.

Issue: On W10, the folder appears “Up to date” but with the Synology device “Syncing” at 0% for the whole day. On Synology, the folder appears “Out of Sync”, with the W10 device “Syncing” at 99%.

Screenshots:

Question: How can I debug the situation ? As far as I know, I have no particular filtering set up on either device, and the folder on Synology is R/W by user, group, and others. I guess this sounds like a newbie question, but I’m a Syncthing newbie :slight_smile:

Any hint would be greatly appreciated.

Vincent

Click on the failed items to see why they failed. Likely guess is a permission error. Permissions on Synology are tricky. There are some notes on setting ut up at https://docs.kastelo.net/synology/sharing/.

Hi Jakob, and thanks for the reply.

That was it indeed! I followed the steps on the page you linked (maybe a link to it in the FAQ would make sense). I changed the settings and within seconds, I had 18175 synced files :-). Perfect.

… or I should say almost perfect, because on the PC, the Global State is 18250 files (75 more thus). Any idea how I can find where the difference comes from ?

… and on both devices, the “Out of Sync Items” line still lists 25 files, just like in the second screenshot above. But the list is the same on both devices. Should I worry about them if the folder shows a green “Up to date” on both sides ?

KR,

Vincent

There are a lot of threads right now on this weirdness. I think something is a bit broken, but it might be with just the GUI accounting if the files are in fact here…

Hi:

Just a quick note:

  • There’s a typo in here:

It is also a good idea to set up some initial ignore patterns at the same time. Synology by default creates a directory ***@aeDir that contains

  • You should also add #snapshot to the list, in case you mount BTRFS folder snapshots through the Synology UI.

Thanks.

1 Like

Hi,

Nice catch. I had already the correct eaDir spelling though, because I’m used to those files (and it’s correct in the screenshot on the page).

I’m not using BTRFS, but good to know.

Thanks.

I have to admit I find it a bit worrisome…

I have added a new folder to sync and the number has risen to 259 “out of sync” items :frowning:

Do you know if that known issue only affects the first sync (in which case I can try to “fix” the out of sync items by hand, e.g. by moving them out and back in the folders), or do they come back randomly? The number of synced files now is about 50000 on 2 machines, but if I settle on Syncthing for all my sync needs, that will be around 10 times more, spread on 4 machines, and I’m afraid I’ll soon overlook those warnings and my family and I will end up having an wrong feeling of safety.

Can I help narrow down the issue ?

KR,

Vincent

Edit: Even more weird: the “Device” section mentions 259 “out of sync” files, but on the new “CDRom” folder alone, it says 656 items are “out of sync” (?):

If I click on the link on those 656 items in the “CDRom” folder, I’m getting this:

The folder and remote device out of sync displays are independent - it’s totally fine for them to show different numbers. The folder one shows how many files need updating for the local device to get up to date, the remote one shows how many files the remote device needs.

In your case the folder is “send-only”, so this means the remote changed those files and it refuses to accept those changes (because it’s send-only). They should go away when you click override changes, but beware that means the remote will get their changes overwritten by what the send-only device has.

Hi imsodin, thanks for your reply.

Makes much sense. I checked that the changes did not matter and chose to override them, and now the 656 “out of sync” items in the CDRom folder have disappeared. Great.

Regarding the 259 items on the right of the PC UI, if I understand correctly, it means that the remote (NAS) needs them.

But what’s strange is that on the NAS UI, it also says 259 items are out of sync (and yes, they’re the same):

Any idea how to solve this issue ?

Kind regards,

Vincent

Anything special about those items? Does the case of the filenames on disk differ (e.g. foobar vs fooBar)?

Hmmm, no, the files all originated from Windows, so two files with different case in the same folder is impossible. Could I find the reason for unsync’d files somewhere in the logs ?

Anyway, I performed two tests on two set of files to try and resync.

  1. For the Audio folder, I moved the offending files out of the source (Win) dir, then forced a resync. Syncthing proposed to override (which I did), and remote files were deleted. I then brought back the files into the original Windows folder and a second sync uploaded the files fine (they disappeared from the “out of sync” list on both sides).
  2. For the CDRom folder, I acted differently : I deleted a set of files listed as “out of sync” from the NAS. I then forced a resync and the files did reappear on the NAS, but they are still listed in the “out of sync” list on the NAS UI.

In this case, I don’t understnand how they could be “needed on the NAS”, because syncthing on the NAS just redownloaded them…

What is the recommended way to bring two folders back in sync ?

Plus, I still have plenty of unsync’d files. If I can grab logs or something that might help understand where the issue lies, I’d be glad to help.

Of course, it could also be yet another usage error on my part :wink:

KR,

Vincent

If the local/global state in the expanded folders match, the folders are likely in sync and “just” the remote device accounting is incorrect.

For further analysis why, please post the output of https://docs.syncthing.net/rest/db-file-get.html for one of the out-of-sync-items (curl example: https://docs.syncthing.net/dev/rest.html#api-key, windows howto: [ How To ] call the REST API from Windows PowerShell).

Indeed, local=global, which is good :slight_smile:

However, I just noticed that the numbers for the “Audio” folder (which shows no “out of sync” files on either side anymore) are different on Win (18 250) vs NAS (18 275). Mmmh 25 files more on the NAS ? Weird.

Thanks for the very helpful links. Here’s the JSON result for one such “out of sync” files:

From Windows:

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2014-01-20T23:20:14+01:00",
    "modifiedBy": "HQMIU6O",
    "mustRescan": false,
    "name": "Raspberry Pi\\magPi-FR\\The-MagPi-issue-8-fr.pdf",
    "noPermissions": false,
    "numBlocks": 117,
    "permissions": "0644",
    "sequence": 32846,
    "size": 15229723,
    "type": "FILE",
    "version": [
      "HQMIU6O:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2014-01-20T23:20:14+01:00",
    "modifiedBy": "HQMIU6O",
    "mustRescan": false,
    "name": "Raspberry Pi\\magPi-FR\\The-MagPi-issue-8-fr.pdf",
    "noPermissions": false,
    "numBlocks": 117,
    "permissions": "0644",
    "sequence": 32846,
    "size": 15229723,
    "type": "FILE",
    "version": [
      "HQMIU6O:1"
    ]
  }
}

Same request from the NAS:

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2014-01-20T23:20:14.1065494+01:00",
    "modifiedBy": "7PUJGCV",
    "mustRescan": false,
    "name": "Raspberry Pi/magPi-FR/The-MagPi-issue-8-fr.pdf",
    "noPermissions": true,
    "numBlocks": 117,
    "sequence": 4179,
    "size": 15229723,
    "type": "FILE",
    "version": [
      "7PUJGCV:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2014-01-20T23:20:14.1065494+01:00",
    "modifiedBy": "7PUJGCV",
    "mustRescan": false,
    "name": "Raspberry Pi/magPi-FR/The-MagPi-issue-8-fr.pdf",
    "noPermissions": true,
    "numBlocks": 117,
    "sequence": 4179,
    "size": 15229723,
    "type": "FILE",
    "version": [
      "7PUJGCV:1"
    ]
  }
}

I admit I don’t see much difference… Do you ?

KR,

Vincent

The versions are different. The NAS should show the file as out of sync on the folder. Can you get the stindex tool from https://build.syncthing.net/buildConfiguration/Syncthing_Tools/65961?buildTab=artifacts and run stindex -mode idxck /path/to/index-v0.14.0.db to check the db.

Hi,

I had to struggle a bit to find the .db location but I think I found it (please check below and tell me if it’s not the right path). I had shut down syncthing before to make sure not to mess with a live DB.

Here are the results on the NAS:

sudo /volume1/misc/stindex -mode idxck /var/packages/syncthing/target/var/index-v0.14.0.db

Sequence entry missing for FileInfo "Compilation/VLC-build/_05.06.J-coreutils/libiconv-1.13.1-2-msys-1.0.13-dll-2.tar.lzma", folder "dhftj-klqfe", seq 4191
Sequence entry missing for FileInfo "Compilation/VLC-build/_05.06.J-coreutils/coreutils-5.97-3-msys-1.0.13-ext.tar.lzma", folder "dhftj-klqfe", seq 4193
Sequence entry missing for FileInfo "Raspberry Pi/magPi-FR/The-MagPi-issue-7-fr.pdf", folder "dhftj-klqfe", seq 4181
Sequence entry missing for FileInfo "Compilation/VLC-build/_05.06.J-coreutils/libintl-0.17-2-msys-dll-8.tar.lzma", folder "dhftj-klqfe", seq 4192
Sequence entry missing for FileInfo "Compilation/VLC-build/_03.B-02-msys/MSYS-1.0.11.exe", folder "dhftj-klqfe", seq 4198
Sequence entry missing for FileInfo "Raspberry Pi/magPi-FR/The-MagPi-issue-5-fr.pdf", folder "dhftj-klqfe", seq 4180
Sequence entry missing for FileInfo "Compilation/VLC-build/_08.H-09.2-msys-pkg-config/glib_2.18.2-1_win32.zip", folder "dhftj-klqfe", seq 4196
Sequence entry missing for FileInfo "Compilation/VLC-build/_02.F-04-mingw-utils/mingw-utils-0.3.tar.gz", folder "dhftj-klqfe", seq 4197
Sequence entry missing for FileInfo "Compilation/VLC-build/_08.H-09.2-msys-pkg-config/pkg-config_0.23-3_win32.zip", folder "dhftj-klqfe", seq 4195
Sequence entry missing for FileInfo "Raspberry Pi/magPi-FR/The-MagPi-issue-8-fr.pdf", folder "dhftj-klqfe", seq 4179
Sequence entry missing for FileInfo "Compilation/VLC-build/_08.H-09.2-msys-pkg-config/pkg-config-dev_0.23-3_win32.zip", folder "dhftj-klqfe", seq 4194
Folder 2 missing 3 sequence entries: #4179 - #4181
Folder 2 missing 8 sequence entries: #4191 - #4198
Missing need entry for needed file ".sync", folder "dhftj-klqfe"
Missing need entry for needed file ".sync/ID", folder "dhftj-klqfe"
Missing need entry for needed file ".sync/IgnoreList", folder "dhftj-klqfe"
Missing need entry for needed file ".sync/Archive", folder "dhftj-klqfe"
Missing need entry for needed file ".sync/StreamsList", folder "dhftj-klqfe"

Note that it’s the whole file. The “out of sync” file I checked via REST above (The-MagPi-issue-8-fr.pdf) is included with a “Sequence entry missing” but there are only 10 other files listed while the UI says there are 115 “out of sync”… Also note that the missing “.sync” (5 last lines) relate to a previous evaluation of Resilio Sync (which of course does not synchronize those folders anymore). The list of excluded patterns does include “.sync” on both sides though…

For reference, running stindex on the Win PC lists 223 “Sequence entry missing” although the UI says there are 234 “out of sync” files, so we’re much closer.

Does this help ?

KR,

Vincent

Do you still use Resilio in parallel? If so, some ignore patterns are needed, otherwise you will have problems when folder overlaps.

Regarding ignore patterns, Syncthing has no standard ignore patterns that are defined in principle or globally. In any case, I don’t know anything about it. The problem may be that Windows files are synchronized that Syncthing may not be able to delete from the NAS later if you delete them on Windows. Then you have to intervene manually, otherwise differences can occur.

I have also Synology for myself, DS412+, 415+, 1815+ and Synology is known to be something special with the ACLs. For example, the file Thumbs.db, which is a typical Windows file, had proven to be a “classic problem” for me. Without ignore patterns, this is of course synced and is also on the NAS. Therefore I use an entry in the Ignore list (?D)(?I)Thumbs.db, which means that no distinction is made between upper and lower case and the file is not transferred, because is ignored, but can be deleted if the process needs. There are other files that I have defined for all peers, since then it runs well, for example I define

(?d)(?i)Thumbs.db
(?d)(?i).thumbnails
(?d)(?i)ehthumbs.db
(?d)(?i)desktop.ini

The only thing that helps is to examine such things and make tests.

Hi Andy, and thanks for replying.

As I said Resilio does not synchronize those folders anymore (it’s still running for other folders until I decide to switch to Syncthing) but they don’t overlap.

I currently exclude the patterns in the screenshot below, in each folder and on each side: 2020-05-24_0925

I will add your patterns. That could explain the difference in numbers, but not the list of “out of sync” files I guess…

KR,

Vincent

I just did another test to check if those alerts are real: compare Win and NAS folders using FreeFileSync.

The result shows that the file reported as “out of sync” and analyzed above (“Raspberry Pi/magPi-FR/The-MagPi-issue-8-fr.pdf”) is, in fact, in sync.

However, some others are not, and they are not reported anymore!

Remember there were first 259 “out of sync” files on the NAS, and trying to force a resync, I deleted a part of them (one folder) from the NAS and forced a resync on both sides. They disappeared from the “out of sync” list, bringing the number down to 115 files… but they were not actually copied :frowning: :

I think this is the worst scenario: having a file sync system in place advertising green “Up to date” messages while there are actually missing files, as it gives a false feeling of security.

This clearly looks like a bug to me (if there is a permission or other issue, it should show up as an error I guess).

What do you think ? How can I help track down this bug ?

KR,

Vincent

I doesnt see anythink in your discussions here. My experience and is also discussed in the forum several times, to set in each peer “Ignore permissions” checked. So best is to test it.

Maybe read in the linked thread the discussion about.

The sequence entry missing errors are the/a problem. While we don’t know the underlying cause of that problem, there is a fix in v1.6.0 that prevents one way how it can occur. I hope it’s the only way, but given I don’t know the root cause, I can’t be sure.

Now for the files that aren’t synced: Are any of those mentioned in the stindex output? If not, can you get the output of the /rest/db/file endpoint for one of those files?

To fix your current sequence related issues without upgrading, you could run Syncthing once with the STRECHECKDBEVERY=1s environment variable set. However lets first investigate the second, more serious issue, and then go on to this step.