Node disagree about "Global State"

Support
1

Hi, i’ve cancelled my BT Sync Pro subscription as soon as i learned about Syncthing, and i really really love it - but i have an issue with a couple of folders…

I’m running Syncthing on 4 devices now, but the issue only happens between 2 particular nodes.

Node 1 (x86 NAS) receives all backups from various other devices and shares that backup folder with Node 2 (ARM NAS). They are both running 0.11.10.

Node 1 has the following info for the backups folder:

Global State 	204886 items, ~190 GiB
Local State 	204886 items, ~190 GiB
Folder Master 	Yes
Ignore Permissions 	Yes
Rescan Interval 	0 s
Shared With 	node2.local

while Node has this info for the (shared) folder:

Global State 	203478 items, ~173 GiB
Local State 	203478 items, ~173 GiB
Ignore Permissions 	Yes
Rescan Interval 	0 s
Shared With 	node1.local

The first node used to have a rescan interval of 1 hour, but i switched to synthing-inotify, so don’t worry about the interval being 0. The second node only receives and is never touched by anything except syncthing, so i figured it doesn’t need a rescan interval, either. I did click “Rescan” for that folder on both boxes, just to make sure.

Anyways, as you can see they disagree not only about their Local State (which is correct), but also about their “Global State” - which shouldn’t happen, right? It’s like this for days already, so it’s not like they just didn’t catch up yet. Node 1 keeps on receiving backups, but Node 2 never hears about those new files…

Any hints would be greatly appreciated, i’d hate to switch back to BT Sync! =(

2

Make sure all devices are connected, and just rescan on all of them. Also, make sure that the list of ignored files is the same on all devices.

3

Both are on the same LAN, and, as i already mentioned, i also did a rescan before posting, just to make sure it’s not that they don’t see the new files.

But rescanning shouldn’t be the issue anyways, as they both do see the exact amount of files they have in their local folders - the non-master node just doesn’t notice that the Global State is actually different from what it thinks it is!

4

Check the ignore files and logs, on the node that is missing files, as some files might get dropped/ignored due to having invalid characters in the name etc.

Also, running with STTRACE=model env var set will show index information arriving from remote peers which might give out some clues to what’s happening.

5

There a no ignore patterns, and the files have only ASCII-based names (always a good idea for backups). Files that were already in that folder (just with different timestamps) get indexed on Node 2, and of course also on Node 1, that’s why it has a “bigger” Global State.

You’re one of the expert, but i really can’t see what this has to do with the files, as everything that’s present in each folder gets picked up by Syncthing.

Node 2 just doesn’t see that the Global State is different from what it knows… and isn’t the Global State independent from the Local State? Don’t the nodes talk to each other to agree on what the actual Global State is?

I’m running it with the STENV=model env var now, getting lots of “sendIndexes” but nothing alarming yet… could take a while.

6

Are you sure you’re not running into this issue: https://github.com/syncthing/syncthing/issues/1751

Look for messages complaining about ping-timeout. Especially low-performance devices (NAS) seem to run into this issue.

7

Thanks, that is a very probable issue! Node 2, the ARM NAS, is mostly at >90% CPU Usage, so i guess that explains why only some files get synced, but most don’t.

8

The solution/ workaround seems to be rather complicated unfortunately. A not-so-quick and dirty fix is to change the time-out threshold. It’s hard coded though, so you will have to compile your own version - on all nodes that are involved

9

This will be fixed in the next release.

10

The new version 0.11.11 with its adjustable ping timeouts works fine after i increased the values by a factor of ten - rather too high then “probably high enough”! :wink: