Out of sync items from removed device

[srottem (Symon)]

Hi, I have a network of devices that are syncing with each other but are reporting many out of sync items and I can’t seem to get rid of them.

First a little history:

Originally the network consisted of devices named nas-fr, nas-au, srottem-toshiba, shornsby-toshiba. There were folders shared between all of the devices which synced as follows:

nas-fr <-> srottem-toshiba
nas-fr <-> shornsy-toshiba
nas-au <-> srottem-toshiba
nas-au <-> shornsby-toshiba 
srottem-toshiba <-> shornsby-toshiba

While disconnected from nas-fr (for about 2 years, pandemic, yay) I added srottem-dell and shornsby-dell so that the network (for a short time) looked like this:

nas-fr <-> srottem-toshiba
nas-fr <-> shornsy-toshiba
nas-au <-> srottem-toshiba
nas-au <-> shornsby-toshiba 
nas-au <-> srottem-dell
nas-au <-> shornsby-dell
srottem-toshiba <-> shornsby-toshiba
srottem-dell <-> shornsby-dell

Once all files had synced to the new srottem-dell and shornsby-dell devices I removed the toshiba devices from the dell devices and nas-au and decommissioned them. That left the network configured like this:

nas-fr <-> srottem-toshiba 
nas-fr <-> shornsy-toshiba 
nas-au <-> srottem-dell
nas-au <-> shornsby-dell
srottem-dell <-> shornsby-dell

It’s worth noting that the toshiba and dell machines can never be connected to both the nas-au and nas-fr devices at the same time as the nas devices are geographically separated are not on the internet.

At this point nas-fr was now very out of date and unaware of the decommissioning of the toshiba devices.

Subsequently the dell machines were connected to nas-fr, the toshiba ones removed and the dell machines synced with nas-fr so the network now looks like this:

nas-fr <-> srottem-dell
nas-fr <-> shornsby-dell
nas-au <-> srottem-dell
nas-au <-> shornsby-dell
srottem-dell <-> shornsby-dell

The problem is that srottem-dell now reports nas-fr as having 6,000+ files out of sync and shornsby-dell 4,500+ out of sync. The out of sync items show the IDs of the removed srottem-toshiba and shornsby-toshiba devices.

Is there any way to resolve this?

[tomasz86]

Can you post screenshots of the Syncthing Web GUI from all affected devices?

Also, are any ignore patterns involved in the problematic folders?

[srottem (Symon)]

Sure, although I can’t give you a screenshot of the nas-au device as it’s on the other side of the world and inaccessible:

All folders have a .stignore file uses an include (#include .stglobalignore) to point to another file that has all the ignore patterns for the folder which looks like this:

// .stglobalignore

// Firefox downloads and other things
*.part
// Chrom(ium|e) downloads
*.crdownload

// Temporary / Backup Files
*~
.*.swp

// OS-generated files (OS X)
.DS_Store
.Spotlight-V100
.Trashes
._*

// OS-generated files (Windows)
desktop.ini
ehthumbs.db
Thumbs.db

// BTSync files
.sync
*.bts
*.!Sync
.SyncID
.SyncIgnore
.SyncArchive
*.SyncPart
*.SyncTemp
*.SyncOld

// Synology files
@eaDir
#recycle
.SynologyWorkingDirectory

The out of sync files, however, don’t appear to conform to anything in the ignore patterns. For example, the following screenshot from srottem-dell illustrates a set of directories and files listed as modified by the removed toshiba devices, which are no longer members of the network:

Those files exist on nas-fr but do not exist on srottem-dell, likely because they were deleted before srottem-dell was added to the nas-fr device. They should no longer exist on either device.

Cheers,

Symon.

[tomasz86]

As the first step, I’d suggest to enable “ignore permissions” in the Advanced tab for all folders. Preferably everywhere, or at least on all NAS devices.

When it comes to “Out of Sync Items”, the size of “0 B” normally indicates that they aren’t modifications but rather deletions. Of course, the problem of them not syncing still remains. Do the actual global/local folder states match between the affected devices?

[srottem (Symon)]

Thanks for the speedy responses!

Yes, they represent deleted files. The files correctly do not exist in srottem-dell, but do on nas-fr. Removing them from nas-fr does resolve the out of sync items for those files, but there are thousands of them so I was hoping for a more generic solution.

I’ve tried enabling the “Ignore Permissions” section on both srottem-dell and nas-fr - no change, unfortunately.

Cheers,

Symon.

[srottem (Symon)]

Actually, checking more closely, the above only applies to some of the files. There are others that don’t appear to exist on any of the devices that are currently registered in the network.

[tomasz86]

Before doing anything more drastic, I’d still suggest to enable “ignore permissions” everywhere, including Windows, and make sure to do it for all folders.

I’ve also just noticed that your ignore patterns lack the case insensitive (?i) in front of them while some of your systems, i.e. the NAS devices, likely run Linux (if not, please specify), and as such they’re case sensitive. I’d suggest to add (?i) to all the patterns, e.g. (?i)desktop.ini instead of just desktop.ini, etc.

Lastly, I was asking about ignore patterns specifically, because there’s a known issue about listing ignored files in “Out of Sync Items” (see https://github.com/syncthing/syncthing/issues/7474), which could potentially affect you, as your devices are only partially connected with one another.

[srottem (Symon)]

Thanks. Yes, the nas hosts are running Linux while the dell machines are Windows. I’ll adjust the ignore patterns as you suggest, but I’m not sure they’re going to be a problem as-is since the normal naming conventions of those automatically generated files tend to conform to the cases as currently specified.

I have enabled “Ignore Permissions” for all folders on the srottem-dell and nas-fr and nothing has changed.

[srottem (Symon)]

I’ve managed to work around the problem by removing all the files that should have been deleted from nas-fr using robocopy from one of the other devices that contained the correct set of files and directories, so my issue is now resolved.

Thanks for the help!

Cheers,

Symon.