synchronization stalled or not progressing or executing

Support
1

Hi,

I’ve a strange issue, synchronization seems to halt, even when not in sync, there is no progress to see in the console window.

This is the setup

My “Master” is a win 10 machine, x64, with syncthing 0.14.5 3 folders defined, each from a network share of a NAS, each defined as master

My “other” node is also a win 10 machine, x64, syncthing 0.14.5 the same 3 folders defined, each from a network share of a nas

for now (testing) they are on the same lan

I would like to use it as a secondary backup solution, so the master should sync to the “other” node, the master should override anything on the other node, kind of like a mirror

the 3 folders are “public” “private” and “sw”

on the master, after the initial scan:

  • sw: local state = 47817 items, ~661 GiB
  • private: local state = 42988 items, ~346 GiB
  • public local state = 3669 items, ~148 GiB

what happens: I paused both devices, so the scan could run without interference. I had both nodes do the initial scan, the destination node (“other”) already has most of the files, but there is of course a difference. Then I “unpaused” and after a while, I could see the out of sync items on my destination, and on the master I had the button “override changes”. I clicked it, and almost immediately, I saw the master suddenly become “up do date”. Which in reality is not yet true, as it still had to sync a lot to the destination??? In the console I could see it sending updates to the destination node.

On the destination node, I could see those updates come in, and then … it became quiet… nothing happens in the console, for one of the folders I see Completion for folder “sw” on device x-x-x-x-x-x-x-xis 100% but in the webgui, I see:

Global State 47817 items, ~661 GiB Local State 47305 items, ~642 GiB Out of Sync Items 48040 items, ~661 GiB

something similar for the private folder

only the public folder seems to be fine.

I did check with another sync and comparison software:

  • about 23GB or 657 files need to be added on the destination
  • about 4GB or 183 files need to be deleted on the destination and so on

so here are my questions

  • what is happening with the sync?
  • why does the master show up-to-date while in reality they are definitely not in sync?
  • on the destination, the sw folder never seemed to be finished with initial scan (no date and time for “last scan”)
  • what other options to I have to force an update to the destination, from the master

I restarted the whole process 2 times already (clear the db at both sides, and redo the initial scan) which takes about 2 days to complete

2

Provide screenshots from both sides

3

My Experiences are about the same, I think. I’m now working with it for about a month, but I don’t see any progress in the syncing process after 1%.

Here’s what I have and did so far:

  • Setup: 2 Synology boxes (DS216Play on both sides, DSM 6) syncing over the internet (30MB/s connection on both sides at the same provider)
  • by now: Syncthing 0.14.5 on both sides
  • about 220Gb in 430000 files, all file sizes and types are present
  • both sides have 95% of the files already present in their directories, realized with an offline sync bringing the 2 boxes together, as to avoid (I thought) a lengthy initial sync
  • I choose the office syno to be the master
  • I have set simple file versioning and 3 versions
  • I have set rescan intervals at 20 minutes to avoid overhead
  • After the fist time I tried, I have switched off compression completely, as I see that network is not the bottleneck (there’s hardly traffic)
  • Connection type shows relay (I can’t find how to change that, but also when I put them together on a LAN for 3 days, there was no change in results).

Results at this moment: On the master side (Office Box), all shared directories show ‘up to date’ and the unshared ones obviously show ‘unshared’. This took around 4-5 days to achieve. I had issues with negative percentages also in version 0.14.5), but after the last reinstall and another 4-5 days of scanning, that did not come back (yet). It regularly shows that the remote device is Disconnected (on both sides, but not at the same time, by the way), but not always: sometimes (usually after an hour or so after I start the GUI) it switches to syncing and then it never shows more than 1%. Also not for letting it run for 10-20 days or so. I have switched off the web GUI during that period after reading that this makes a performance difference.

After initial freezes and no progress for >10 days, I have deleted and reinstalled Syncthing 2 times on both sides. Just to try again and see what happens. No changes. Originally, I synced 5 directories, but in the last attempt I have split a few larger directories into smaller ones, trying to keep things manageable. I did scan all directories on the ‘master side’ in 3 days, but I do not yet share them all; so far only about 5 GB in 6 directories, also in an attempt to keep things manageable.

Still, I do not get more than 1% synced…

The CPU and RAM are constantly working at 80-100% levels and there is hardly any network traffic.

Obviously, I’m not happy and impressed by the performance and results. I hope sharing my experiences helps in improving this promising project.

  • Is there any way I can improve performance, more than I already tried?
  • Is there any hope that this process will start up a regular sync on 2 DS 216Play Synology boxes at all, or are they just not powerful enough to deal with Syncthing in combination with more than 1-2 GB of files?

Thanks for any help. I’m very willing to provide more information to solve this; I’m not a developer so keep it simple.

Thanks.

4

syncthing_sample_master.jpg1232×816 131 KB
syncthing_sample_remote.jpg1221×678 127 KB

I uploaded 2 screenshots as asked.

one from the master (the one with folder type “master”) and one of the remote destination.

You’ll see the info on the “sw” folder I discussed in my opening post

5

The state “Up to Date” on the top screenshot is correct because it has all the files, remote state says syncing which is also correct. Bottom screenshot has it the other way round which is also expected.

Bottom screenshot shows scanning. While it is doing this it will not sync anything so you have to wait for that to finish. You could increase the rescan interval (or even set it to 0) so that scanning does not block syncing and try if that helps. “each from a network share of a nas” sounds like scanning could be quite inefficient and block syncing for quite some time.

Another thing: you are connected over relay so transfer speed is likely limited by that and you may get frequent disconnects depending on the relay (and thus get the possibility to start a new scan, it does not do that while syncing).

6

Hi

it is especially frustrating for us, as half of the time, where I would expect syncthing to do something, it does not seem to do anything.

for example the snippet of logging below:

to me, it it does not seem to do anything between 01:55:13 and 05:04:23 also, at 01:55:13 it says “Completion … 100%” while that folder still shows out of sync for about 20 GB’s.

so if we could see what syncthing is doing from 1:55 till 5:04, it would help alot against the complaints and frustration.

for example, walking/scanning the directory structure … that seems to take a few hours, EVERY (scan) time? but we don’t have any indication of it

[SLUKD] 2016/08/30 01:55:03 VERBOSE: Finished syncing "private" / "data\\stock-photowork\\xxxxxx.jpg" (metadata file): Success
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Started syncing "private" / "ced-material\\xxxxxxxx.avi" (metadata file)
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Finished syncing "private" / "ced-material\\xxxxxxxxx.avi" (metadata file): Success
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Started syncing "private" / "ced-material\\Utilities\\xxxxxxxx.exe" (metadata file)
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Finished syncing "private" / "ced-material\\Utilities\\xxxxxxxxx.exe" (metadata file): Success
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Started syncing "private" / "ced-material\\Virtual Machines\\xxxxxxxx" (metadata file)
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Finished syncing "private" / "ced-material\\Virtual Machines\\xxxxxxxx" (metadata file): Success
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Started syncing "private" / "data\\stock-photowork\\_wallpapers\\xxxxxxx.jpg" (metadata file)
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Finished syncing "private" / "data\\stock-photowork\\_wallpapers\\xxxxxx.jpg" (metadata file): Success
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Started syncing "private" / "ZNo\\xxxxxxxxx" (update file)
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Summary for folder "private" is map[globalBytes:369938885774 globalFiles:42992 needFiles:1695 globalDeleted:14128 localBytes:364234295206 inSyncFiles:41297 state:syncing localDeleted:0 needBytes:43665605872 inSyncBytes:326273279902 sequence:188055 localFiles:42994 version:188055]
[SLUKD] 2016/08/30 01:55:03 VERBOSE: Completion for folder "private" on device x-x-x-x-x-x-x-x is 100%
[SLUKD] 2016/08/30 01:55:11 VERBOSE: Finished syncing "private" / "ZNo\\xxxxxxxxx" (update file): Success
[SLUKD] 2016/08/30 01:55:13 VERBOSE: Summary for folder "private" is map[inSyncFiles:41351 needFiles:1641 state:syncing globalFiles:42992 globalDeleted:14128 localDeleted:0 needBytes:43024295520 localFiles:42995 localBytes:364254896038 globalBytes:369938885774 version:188078 sequence:188078 inSyncBytes:326914590254]
[SLUKD] 2016/08/30 01:55:13 VERBOSE: Completion for folder "private" on device x-x-x-x-x-x-x-x is 100%
[SLUKD] 2016/08/30 02:06:08 VERBOSE: Folder "public" is now scanning
[SLUKD] 2016/08/30 02:07:10 VERBOSE: Scanning folder "public", 0% done (0.0 MB/s)
[SLUKD] 2016/08/30 02:07:45 VERBOSE: Folder "public" is now idle
[SLUKD] 2016/08/30 02:56:17 VERBOSE: Folder "public" is now scanning
[SLUKD] 2016/08/30 02:57:21 VERBOSE: Scanning folder "public", 0% done (0.0 MB/s)
[SLUKD] 2016/08/30 02:58:01 VERBOSE: Folder "public" is now idle
[SLUKD] 2016/08/30 03:55:15 VERBOSE: Folder "public" is now scanning
[SLUKD] 2016/08/30 03:56:21 VERBOSE: Scanning folder "public", 0% done (0.0 MB/s)
[SLUKD] 2016/08/30 03:57:00 VERBOSE: Folder "public" is now idle
[SLUKD] 2016/08/30 04:43:07 VERBOSE: Folder "public" is now scanning
[SLUKD] 2016/08/30 04:44:11 VERBOSE: Scanning folder "public", 0% done (0.0 MB/s)
[SLUKD] 2016/08/30 04:44:49 VERBOSE: Folder "public" is now idle
[SLUKD] 2016/08/30 05:04:23 VERBOSE: Started syncing "private" / "ced-material\\_eBooks\\xxxxxxxxx.pdf" (metadata file)
[SLUKD] 2016/08/30 05:04:23 VERBOSE: Finished syncing "private" / "ced-material\\_eBooks\\xxxxxxxxx.pdf" (metadata file): Success
[SLUKD] 2016/08/30 05:04:23 VERBOSE: Started syncing "private" / "ced-material\\_eBooks\\xxx" (metadata file)
[SLUKD] 2016/08/30 05:04:23 VERBOSE: Finished syncing "private" / "ced-material\\_eBooks\\xxx" (metadata file): Success
[SLUKD] 2016/08/30 05:04:23 VERBOSE: Started syncing "private" / "ced-material\\xxx" (metadata file)
[SLUKD] 2016/08/30 05:04:23 VERBOSE: Finished syncing "private" / "ced-material\\xxx" (metadata file): Success

P.S. I know sync speed will be slow over the internet, I don’t mind that, it is more the problem of not seeing what syncthing is doing, while it seems idle to us

7

It looks from your screenshot that on one side you are sharing 3 folders with the other device, and on the other side just one of the three, hence 33% completion.

8

Hi,

no, I’m sharing and syncing all 3 folders now (today at 19:00) the master shows syncing (78%) to the remote node on the remote node the public and private folder are now “up to date” while the sw folder is still syncing (63%)

It was syncing this morning until 08:52, then it stopped for which reason? (nothing in the log, and it was not rescanning folders, because the last scan was at 08:44 this morning. It started again at 10:44, stopped at 11:39, did nothing until 16:20, then it started syncing again, and so on… So why is it stopping, or what else is it suddenly doing? Restarting does not help.

So it still comes down to visibility to start with : what is syncthing doing while it does not show anything in the log?

All the basics are fine : network connection, connection to the nas, connection to the internet, because my other backup system keeps going, without stopping

9

You can run syncthing with -verbose -audit flags and post the logs for all devices.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.