Both Remote Devices Reporting "Out of Sync" by about 750k Items

Running ST 1.23.4 as service on 2 x Windows 2022 Servers with 15 or so folders (almost 5TB) which are mostly reporting “Up to Date” or within 20 files since it’s actively used. But the Remote Device is reporting that 748,000+ files are Out of Sync on both sides. We recently migrated from a perfectly working 2 x 2012 R2 ST setup (0 out of sync items), to a 2 x 2022 setup. Very little progress (few hundred items reduced).

We are syncing Permissions and previously did on our 2012 R2 setup and are using the same service account on the new 2022 server. I checked the db logs, and see they’re 3 hrs behind (2k+), but the log file is up to date. See screenshot. It’s syncing both ways, since I tested adding/changing/deleting folders and files on both sides and it syncs.

What I did:

  • I restarted ST on both sides, and still the same issue after 3 hrs.
  • Checked the ST log, and nothing related to this, just some minor sync errors due to files/folders with problems I know about.

Any recommendations on where to look? Thanks

Have you checked the list of 748,541 out of sync items to see if there’s a notable pattern?

Enabling the syncing of permissions comes with caveats. Are the pair of Windows 2012 R2 servers the same ones that are now running Windows 2022?

I don’t see any pattern yet, but I didn’t realize I could review the list. I checked a # of files, and NTFS shows Full Control for the service account. (I see the new versions have improved this UI tremendously). Nice! ST is running as a service too. Could there be a service restriction?

I realize syncing permissions has caveats. I’ve been running with ST w/syncing permissions for probably 4-5 years w/o issues. It was ROCK solid (can’t say the same for Resilio which is what forced us to switch).

Brand NEW VMs. These are brand new Server 2022 VMs (clean OS install). We robocopied the files from NY 2012 R2 to NY 2022 and TX 2012 R2 to TX 2022 file servers VMs and then enabled/setup ST. On our 2012 R2 VMs we were running ST via SyncTrayzor, not we’re running ST via service account.

Performance: New servers have 4x the memory (64GB), 3x the vCPUs (12 vCPUs), and it’s pure SSD storage now. CPU architecture is 4-5 generations newer too. The new servers should fly.

Any other ideas? Thanks!

Yes, the service account is privileged, but isn’t an administrator.

Although it’s possible to run Syncthing via the service account, it’s safer in terms of security and less messy in terms of management to run Syncthing as the same user who owns the files being synced.

For sure, switching to a service account added some complexity that needs to be sorted out.

I think the other wrinkle is that the files were copied from one server to another. Under the hood, Windows’ user security model is based on a SID, so two accounts with the same username on different computers aren’t the same users.

What how and what options did you use for Robocopy?

Given that both systems are Windows, it’s likely safe to rule out filename issues (unless those two servers are providing network shares to non-Windows clients).

I’d verify that the out-of-sync items have the necessary permissions, including any subfolders they’re located in, for Syncthing to be able to access them.

Check out Microsoft’s Sysinternals utility suite. In particular, the AccessChk and AccessEnum tools.

I should have been more clear, when I mentioned service account, it’s a service based ST but using the same previously working account that SyncTrayzor was using. It’s not a permission issue.

I did notice that many of the files were tagged with “block” file attribute that Windows thinks that the file was downloaded from the internet. Pic shows what it would say.

ST Unblock File

Files were copied between servers (old Win to new Win Servers) via robocopy. I asked the tech who handled that for the parameters used, but no response yet. So, File Properties, would show “Unblock” as an option. Ran the Microsoft PS command to unblock. recursive across the 1.5m files, but it didn’t fix ST out of sync issue.

Finally removed both folders and waited 5 minutes until the Out of Sync went to 0. Re-added the folders and it’s counting down (completed about 27%). Currently 1.1m files left from 1.5 million. Fingers crossed!

Success. As of Sat AM (about 2 days), it’s fully synced (3.51TB, 1.5m, 481k folders, SSD servers, Gb connection). I suspect the blocked file blocked ST and caused some issue with re-syncing.

Fix: Removing the block on the file attribute and then removing and re-adding the folder was the fix.

@gadget Thanks!

1 Like