First Sync- 2,446 Items out of Sync (all the directories?)

Support
1

Re-trying Syncthing after a gap of maybe 5 years. Initial setup is quite simple:

  • Two Windows 10 machines on a LAN
  • Each machine has a drive (D:) set up for files to be synced
  • Syncthing set up to run as a service

Set up a Syncthing folder share as follows:

  • Label: Shared (common)
  • Path: D:\Shared
  • Versioning: Off
  • Ignore Patterns: (none)
  • Folder Type: Send & Receive (both machines)
  • Advanced options: Sync Ownership

On both machines the user account SyncthingServiceAcct has Modify permission applied at the root of the D: drive.

The Shared folder is quite large, comprising 39,248 files in 2,446 folders.

The Problem

At the end of the initial sync the folder status showed:

Global State 39,248 2,446 ~487 GiB
Local State 39,248 2,446 ~487 GiB
Out of Sync Items 2,446 items, ~306 KiB
Failed Items 2,446 items

The number of failed items is suspiciously exactly the same as the number of subfolders in the sync set, so I assume this is where the problem lies.

Looking at the log on the sending machine I see a long stream of errors of this form:

2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "music\\Library\\Snow Patrol\\Eyes Open"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "music\\Library\\Various Artists\\The Best Of Movie Tunes"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "photo\\~Incoming\\Me\\2005-01 PreSchool Teachers"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "photo\\~Incoming\\Me\\2008-02\\CaptureOne\\Cache\\Proxies"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "archive\\Software\\Windows Apps\\Apps+Utilities"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "archive\\Pictures\\Masked Images\\w_acc"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "music\\Library\\Lynyrd Skynyrd"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "archive\\Admin\\Win10\\ProgramData\\_scripts\\.git\\objects\\27"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "music\\Library\\Meat Loaf"): syncing: handling dir (setting metadata): Access is denied.
2024-10-19 19:36:22 Puller (folder "Shared (common)" (icdyz-xd4y5), item "music\\MP3\\Funeral for a Friend"): syncing: handling dir (setting metadata): Access is denied.

The Syncthing docs say that giving the Syncthing service “Modify” permissions is sufficent, but clearly there is a permissions problem here.

Using the “Advanced Security Settings” on the root of the D drive on the receiving machine, if I check the Effective Access for the SyncthingServiceAcct user I see a red cross (no permission) for the following items:

  • Full control
  • Delete subfolders and files
  • Change permissions
  • Take ownership

My understanding of Windows ACLs is sketchy, I am more familiar with Unix permissions, but I can’t see how the Syncthing service can do what it needs to do to maintain the synced folders WITHOUT the last 3 of the above 4 permissions.

So, is the documentation misleading. Does SyncthingServiceAcct really need Full Control access to the synced folders to be able to copy not just data but also ownership and permissions between devices?

2

Based on what you’ve posted it looks like the SyncthingServiceAcct needs “Change permissions” —- but if I’m wrong then in your case I would use “Full control”.

3

I used the advanced permissions in Windows to add Delete subfolders and files and Change permissions perms for the Syncthing user on both machines.

However a rescan on the sender doesn’t seem to do anything. The sender is stuck on Out of Sync, while on the receiver the Shared folder is shown as Preparing to Sync.

My guess (and it is only a guess) is that the files on the sender and receiver differ only in terms of metadata, the files themselves and checksums are valid, therefore the rescan is skipping all the files.

I don’t see an option to do a “Rescan --force” type operation.

Is thre a way to get over this and force the receiver to itry setting the metadata properties again?

Otherwise there’s the brute force method of breaking the sync connection, deleting all files (486GB of them…) on the receiver, and then re-making the sync connection, but that’s not very attractive.

1 Like
4

You could try starting Syncthing with --reset-deltas, and if that doesn’t work, then with --reset-database. Please keep in mind that resetting the database will force Syncthing to rehash all files in the folder.

1 Like
5

One update- It looks like the timed scans are running. Those are only one per hour by default, but if i drop the rescan interval down to 5 or 10 minutes hopefully I will be able to do testing at a reaonable rate.

I’m still getting streams of permissions errors in the logs, so there is still a problem. I will have to do some research reading.

6

I did a full “clean start” of this sync job with a new ID.

I also did everything I could think of to make sure the synced folder structures on both machines have clean + consistent file owners and permissions, with myself as the owner (I do have admin rights) and with “Full control” permissions for SyncthingServiceAcctinherited from the top level folder of the synced set.

If I run this with “Sync ownership” turned off, the folders do eventually report as being in-sync.

However, if I turn on “Sync ownership” on both machines, once the rescan has finished the status changes to “Out of sync”, although the strange thing is the number of “failed items” reported is different.

Machine 1 reports 2,225 items out of sync, where 2,225 is the number of folders in the sync set.

However machine 2 reports 39,047 items as failed, where that number is the count of files in the set, not folders.

On machine 2, the detailed list of failed items shows errors of the following forms:

For files

syncing: shortcut file (setting metadata): Access is denied.

For folders

syncing: handling dir (setting metadata): Access is denied.

So, despite my best efforts Syncthing still seems to be running into permissions errors when it tries to synchronise the metadata (presumably for ownership+permissions).

Meanwhile, why it’s generating errors about shortcut files I have no idea. There may be a few windows shortcuts in this large data set, but it’s certainly not the case that EVERY file is a shortcut!

Is there a debug flag that will help understand exactly what is failing? I can’t tell whether the “fs” and/or “walkfs” debug flags are relevant, or whether something else is needed.

7

Are the files and folders stored under D:\Shared owned by a mix of different Windows users?

8

No, at least not willingly. I’ve done whatever I could to make the permissions clean and consistent.

I used the advanced permissions to-

  1. Force a single consistent owner (me) on the whole share
  2. Add “Full Control” permissions for the SyncthingServiceAcct user at the root of the share
  3. Forcibly replace all the permissions underneath the root folder with “inherit”

Given the permissions I have set on this folder hierarchy, I can’t understand how anything could be preventing ST from writing file data as it needs, it has "Full control"on everything.

TBH the propagation of permissions through Syncthing seems to be fraught with issues. I know i’m not alone in having problems, there are many threads on the ST and other forums about this.

I’ve seen mentions in several threads of just turning off “Sync ownership”, and having Syncthing write new files+folders with permissions inherited from the folder where the file is being created.

This is less flexible but for my purposes on a simple home network I may be able make it work, at least as a workaround while I grapple with the permissions problems of Syncthing itself.

In some of these discussions i’ve seen this referred to as a “new option” that can be set, however those threads were quite old. I see no option for this in the web UI. Is it supposed to be the default behaviour now?

Some other threads describe setting the umask on the Syncthing daemon process, but umask is a Unix/Linux thing. I don’t know if there’s an equivalent way of controlling the default permissions on files+folders created by Windows Services?

9

You almost certainly don’t want to use “sync ownership”, and enabling it without jumping through the requisite hoops will cause permission errors. If you did intend to use it you must run Syncthing with some sort of system privileges that allows it to change the ownership of your files.

10

You almost certainly don’t want to use “sync ownership”, and enabling it without jumping through the requisite hoops will cause permission errors.

If that is the case-

  1. Is there an option (maybe only in a config file?) to get Syncthing to write subfolders and files with permissions inherited from the containing directory?
  2. How does that work for files inheriting from a “template” set of permissions on a directory? (e.g execute permissions means something quite differnt on a file vs. a folder)?

This first example i’m working on is really simple. It’s a single folder structure of “shared stuff” and I can get away with a single permissions setup for everything.

However if/when I get this working, the next thing I want to do is synchronise multiple users personal files. That is, not $HOME on Linux or C:/users/ on Windows, but the “XDG” dirs (Documents, Pictures, Videos etc.) within the home folders.

For this, if I can’t reliably sync permissions across machines, I would need to set it up so that anything written by Syncthing into (say) D:\Personal\<username>\Documents inherits the ownership and permissions of the Documents directory it is under.

Can Syncthing do that?

11

I believe this is the default behavior on Windows. Essentially the only permission Syncthing sets on Windows is the read only bit.

I don’t know what this means, sorry. I don’t know what execute means on a directory on Windows, but I’m pretty sure Syncthing doesn’t set it. The execute permissions for files, such as it is, is not set either, but is typically inherited and doesn’t work the same way it does on unixes.

Yes, if you enable Sync Ownership and make sure Syncthing has the rights to modify ownership, whatever that means on Windows (maybe running as system or something, I’m not entirely sure).

12

I thought i’d done that by giving the SyncthingServiceAcct “Full Control” privileges, but maybe there’s some extra obscure Windows privilege involved.

13

Yes, the two are different. As a user A with full control, I can’t normally change the ownership to a different user B.

14

I went back over some of the history of the discussions around this issue.

I have some more understanding of the position taken by the Syncthing maintainers. However this does still leave me with the problem of my user data files gradually accumulating cruft in terms of unexpected file ownership and permissions. These are potentially exacerbated as files are propagated between machines.

Now I understand more about how Windows deals with ownership and permissions when copying files, plus the fun of mapping ACLs and permissions when a Linux machine is added to the mix, it seems almost inevitable that these issues will arise.

Up until now I have been using a Linux+Samba machine as a central filestore, and the Samba force user and force group option, plus umask, have provided a fairly simple way of ensuring that anything (for example) than can be written into the server folder of user X, ends up owned by user X with sane permissions on the server.

I was hoping to move to a more decentralised setup with each PC having a copy of all the user’s personal files, and changes synced so there is a degree of redundancy in case a machine dies. However this won’t be popular if it starts to suffer from “permission denied” errors.

My first stab at a solution is a Powershell script that trawls the user files and enforces a simple set of ownership, permissions, and other rules (e.g. hiding any “.*” dotfiles that encroach from the Linux side). This works but it’s a brute force approach and quite slow. It’s not the time the script takes to run that concerns me so much, but the impact on system resources (especially disk IO) which means it could cause noticeable slowdowns if I run it as a regular task through task manager.

I suspect a solution with less performance impact could be put together using something like Watchman (Installation | Watchman) to monitor filesystem changes, feeding a “rules enforcer” written in a more efficient language than Powershell. I’m just more sure if my rudimentary programming skills are up to writing something like that.

If anyone reading this comment has ideas on how to easily create a “rules engine”, or even (ideally) can point me to one that already exists, that would be interesting to hear.