Syncthing does not sync and is stuck

Support
1

Hello to the Syncthing team,

This post may not be pleasant to read but I believe it will reflect the feeling of many lost users. Syncthing would be a great application if I could get it to work. I found the concept so convenient and powerfull that I spent DAYS… exploring your forum, youtube, google to try to get it to work but with no success. I did not want to give up. I tried really hard.

I see many problems:

  • Documentation jumps from the “Quick start guide” to some very detailled “parameters, files, etc…”. For example, I did not find anything mentionning in the documentation how works the “Incoming rate limit”. Apparently it does not apply to local network. But I understood that after reading pages and pages on the forum…
  • Questions on forums often do not lead to a ‘Solved’ conclusion.
  • Support community just acts as if everybody would know everything of this tool by heart. The answer could be “Press the override button” but will not say where to find the override button. Then good luck. Tell me, the vocation of this tool is to be used by anybody, right? not only (Linux|Windows)&Networking experts.
  • This tool is free and maintained by people doing their best. Thus I should not complain too much. Agreed. However, time is money and if this tool appears to be a trap because it requires high technical skills, then this should be printed on the home page. And on forums, you cannot finish a discussion by “If you are not happy, feel free to help!”. Syncthing users could be people who want to sync two PCs, have no skills to help and did not think they would spend some much time trying to get it to work.
  • Some discussions finish by very technical recommendations that are almost impossible to understand to everybody (and even to engineers like me…). Answerers should consider that they are talking to idiots (like me).

After saying that, if you still accept to help, here is my problem:

I trying to sync 3 PCs: 2 under winwows7 and 1 under Vista. 5 shared folders with very large contents (1 folder is 191 Gb). Some of the folders (the smaller ones) synchronised. Not very quickly… but after fews days. The largest folders do not sync or (maybe) progress very very slowly. Some source folders are on 1 PC and do not replicate to the other PCs. Some other source folders are on another PC and do not replicate to the other PCs. (So this does not seem to be specific to one PC, for example the Vista PC).

I though, based on forums, that it was related to the UI interface running, then Firewall, Avast, relay, etc…

Then, in the console window, on a regular basis, I noticed the following message on hundreds of files: [MRDTA] 15:15:00 INFO: Puller (folder “SYNC DOCS PARTAGES” (xxxxx-yyyyy), file “toto.mp3”): finisher: pull: peers who had this file went away, or the file has changed while syncing. will retry later". In the replicating directories, I have hundreds of temp files: ~syncthing~myfile.JPG.tmp

I did not change any files in the source directory.

From the forums, I understood that I should “touch” the source files. There are thousens of them and on Windows, this is not easy to do.

For now, 80 Gb of the 190 Gb have been synced. But I am thinking of stopping the folder sharing and restart with a new folder. Do you have any suggestions. If yes, remember that I am an Idiot :wink: Best regards,

Mathieu

2

Try this (this is violent but that can’t hurt) : on each 3 devices, Remove one common folder share. Once done, in each device, remove all temp files in the share then once done, recreate the share on only 2 devices. If it starts synching, nice, then go on and recreate the share on the 3rd device. This is not an explanation of what happened but I once did it and it worked.

Don’t forget ST version is still 0.xx.xx

3

Hi Mathieu,

concerning your technical unrelated remarks you have my full understanding. To make it short: Don’t give up, keep asking, keep reporting! I’m in this business for some years now ( 4\d ) and I keep telling developers “Try to explain this to your mom||dad”.

Try to consider yourself on the other side. Devs don’t want to explain, they want to code. Any minute they waste by explaining the obvious ™ is a minute less spent on coding.

So don’t give up. Ignore replies like “you better send an PR than complain”. ( whatever a PR is ).

BUT! ( and now we come to your technical problem )

You are asking a question which raises a couple of returned questions. So even if a tech/dev was willing to answer your part he has to ask you questions. ( rewind to time wasted ). Just keep this in mind.

  1. What network speed is between the 3 hosts. 100MBit, 1GbE, WLAN ? How long will it take you to copy a 10 GB file from A to B ? This is essential to decide if your issue fails on physics or on a software problem.

  2. Is it one folder with one 191 GB file, or 191 files á 1 GB or 191.000 files a one kB ? I’ve seen all of this, and all have their own issues.

  3. There is the possibility to create logs. Try to create them, if you fail how to: ask. Ask in a different topic. This way you tell people to “fix their docs” since they seem not to be clear enough.

  4. Your config is also essential. Did you just install syncthing and used the “defaults” or did you tweak anything ? That’s ok, but a dev should know. But back to the logs: They would be more helpful than any explanation.

  5. Logfiles. As said before. Logfiles. There is a saying in the android community: “logcat or GTFO”. ( logcat is the android command to show the logs, GTFO is a nontechnical abbreviation.

See, even I, as a syncthing user, not dev, am willing to help.

3 Likes
4

I meant “the same” one, what means all the ones that have the same folder id. And about the recreation step, do it on a device (pointing to the initial root directory), invite only an other one and wait this one to be prompted, accept invitation, only taking care to fill the Folder path field to the same place it was before you removed it.

BTW, you didn’t say anything on you config : WAN/LAN/Mixed, use of ignore patterns, status in the GUI for other devices (Out of Sync/Pushing files/Yellow titled notification errors). If all seems working but you feel it isn’t, a look at Bytes/throughput counters at [My]-device(s) level would learn you something. System monitor/Task Manager would also show something about network traffic.

5

Hi Cosas, Hi Peter,

Thanks to both of you for your kind answers and encouraging support. Many questions to answer.

  • Configuration was “triangular”.

  • 3 PCs on a home LAN behind an ADSL box. At the begining 2 were on Wifi. But eventually all of them were on Ethernet (cannot tell you the bandwidth).

    PC 1 - Vista:

    • folder A (191 Gb, 204 000 files) shared with PC2, PC3 This folder never replicated completely on PC2 and PC3. The files from this folder were generating errors in the logs: "finisher: pull: peers who had this file went away, or the file has changed while syncing." After my first post, I could see that the replicated folders were still growing but very slowly and the printed transfer rate was from few Bytes/s to few Kbytes/s. I have “Networx” installed on each PC, and I can see that the downstream and upstream activity on interfaces is very very small.
    • folders B, C, D, E, F with source on PC2 or PC3. Some folders fully synchronised (because they were much smaller). Folder B was replicated at 7.5 Gbytes out off the 33 Gbytes.

    PC 2 - Windows 7 - 3 “source” folders B, C, D:

    • 1 folder B (33 GBytes, 88 000 files) shared with PC1 This folder never replicated completely on PC1. The files from this folder were generating errors in the logs: “finisher: pull: peers who had this file went away, or the file has changed while syncing.”
    • folder A replicated up to 98 Gbytes out of 191 Gbytes (after many days),
    • Folders C, D shared with PC1 and PC2. Much smaller (1.5 GBytes and 785 MBytes). Those folders were replicated to PC1 and PC2 (but after quite some time).

    PC 3 - Windows 7 - 2 “source” folders E and F: shared with PC1 and PC2

    • One of the “source” folders was quite large (17.3 GBytes), but eventually replicated on PC1 and PC2 after many days.
    • folder A replicated up to 100 Gbytes out of 191 Gbytes (after many days),
    • folder C, D, E replicated (because they were smaller).

  • Following Cosas’suggestion I have removed all sharing from all PCs. I have removed all .tmp files in half-replicated folders. For your information this was 42 000 tmp files / 20 Gbytes. I restarted the sharing from PC2 to PC1 only. 2 PCs, One way. The 33 Gbytes folder. Replication started quickly, but then went down and after 8 hours, 9 Gbytes were replicated… Is this the normal performances of Syncthing?

  • Following Peter’s question, I installed an FTP Server on PC1 and transfered via Filezilla from PC2 a 30 Gbytes zip file. Transfer rate was 11 MBytes/s and file was transfered in 43 Minutes. Very stable transfer. On networx, it looks like a clean Rectangle.

  • Then I restarted with a brand new folder on PC2 shared with PC1. One way. No other sharing. I populated gradualy the source folder with subfolders.

    • When the subfolder is 100 Mbytes, replicaton takes place very quickly (<10 secs). I can see it on Networx. Bandwidth average is arround 5 Mbytes/s but not stable. Lots of bandwith variation. And going smaller and smaller.
    • When I added a larger subfolder (3.26 GBytes), Bandwidth went down quite quickly. Syncthing is showing a transfer of few KBytes/sec to few 100KBytes/sec. After some time, it shows transfer = O Bytes/s. then resumes a little bit.

  • I have some doute about the Upload rate and Download rate displayed by Syncthing. If the folder to sync is very large, I have the feeling that Syncthing is not able to show a realistic size of the local State. I understand it is hard to assess the size of the remote, but why Syncthing is not able to assess the size of the local folder?

  • When we ask Syncthing to synchronise a very large folder, performances drop drastically until the transfer looks stuck. Transfer rate in Syncthing is 0 Bytes/sec for few minutes. Both PCs are doing nothing else. I checked the CPU in the task manager. No application is taking the CPU. Sometime Syncthing takes 20% but most of the time it is around 0%. For example, at this time, the source folder is 7.24 GB, the target folder is 4.97 GB. Syncthing on the target PC says that remote is 7.13 GB and local folder is 2.19 GB!? Out of sync value is therefore pesimistic and transfer is 0 Bytes/s. Then after… 2 minutes there is a small transfer 5KB/s… then goes back to 0 B/s, etc…

  • Both PC show the same time (5 sec difference).

  • Logs: I almost uploaded the logs on Syncthing console from both PCs. But then, I realised that it shows my PC IDs and folder IDs. I am not sure if this is safe… Please advice on how to extract logs and upload them in the post.

Sorry for the long post. I hope it is useful. Many thanks,

Mathieu
6

Uploading the logs should not be an issue… The IDs are unique identifiers for your machines but aren’t kept secret for any reason. If a machine you don’t know the ID of tries to connect just deny it.

7

@Mathieu_Biskouazic , in troubleshooting similar behaviour, I have found the Windows Resource Monitor to be of invaluable help. Particularly the Disk tab where you will find the Disk Activity panel. Sort the data by column Total (B/sec) to get the most interesting view on things.

Hit Start/Run and type resmon.exe to open up the Resource Monitor.

In there, you might notice things like Syncthing creating the index files, you might notice a HDD not responding properly causing ST to slowdown and even give up temporarily on the indexing process. For sure, untill indexing is complete on all machines, your setup will “struggle”.

Try it out, it is an easy way of getting info (watching Syncthing & other processes at work) without digging through logs. But don’t forget that logs are indeed extremely important once an issue is discovered.

8

Some feedback about my Syncthing configuration.

  • I think that Syncthing more or less works but it is just very slow (transfer rates are far below Windows sharing xfer rates).
  • For me, the ST performances drop when the size of the folders to sync is very large. Maybe because ST scanning/assessment procedure on large folders takes over the actual synchronization task.
  • Because I did not want to wait forever to sync for example my 190 GB folder, I turned off syncthing, used the Windows sharing to transfer (took 7 hours) and replicate the folder on the 3 PCs and then enabled Syncthing afterwards. So for me ST may be ok to maintain synchronization between folders but is not appropriate when used on fully desinchronized (large) folders.
  • I think that the ST GUI is always pessimistic and most of the time shows lower figures than reality. This made me think that ST was stuck when it was actually doing some sync (but still at a very low transfer rate).
  • Syncthing seems not able to synchronise large files (for example 250 MB file).
  • Maybe it is a bit ambitious to display status of distant computers. I often found that the “distant machines” status was wrong. I.e: The GUI is saying that the distant machine is syncing (56% 15 GB) when on the distant machine, the GUI shows that all folders are up to date. Sometime it shows that the distant is not connected whereas it is. The GUI should first concentrate on the local data and:
    • Correctly assess the size of local folders (because it is often not correct…)
    • Correctly display the out of sync info (for the local machine).
  • I Now often run TreeSize on the 3 PCs to verify the actual status of the sync folders.

I will let you know if I encounter issues with this configuration but I need some time to verify if I am not losing data. Thanks for your help.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.