syncthing 0.13.4-1 never makes any progress with syncing

Support
1

Hi all.

Since I’ve upgraded syncthing to 0.13.4-1, one of my machines seems to now get stuck on syncing files. It happens on all the folders. I get messages like the following send to stdout: [LRN43] 15:59:41 INFO: Folder “downloads” isn’t making progress. Pausing puller for 1m0s. [LRN43] 15:59:54 INFO: Folder “sdk_install” isn’t making progress. Pausing puller for 1m0s. [LRN43] 15:59:59 INFO: Folder “work_bin” isn’t making progress. Pausing puller for 1m0s. [LRN43] 16:00:03 INFO: Folder “VirtualBox_VMs” isn’t making progress. Pausing puller for 1m0s. [LRN43] 16:00:52 INFO: Folder “VirtualBox_VMs” isn’t making progress. Pausing puller for 1m0s. [LRN43] 16:01:00 INFO: Folder “work_bin” isn’t making progress. Pausing puller for 1m0s.

These messages get printed repeatedly and no progress is seen on the GUI. Both my systems are using the Arch linux distro the same syncthing version. Is there something I can do to figure out why this is happening? (Apart from rolling back to 0.12 ) ?

Thanks in advance,

2

There should be other messages in the log, which indicate why the puller’s failing?

3

I don’t see any other messages. Is that with the standard log level (without any extra options)? I tried once with -verbose but then the output is really heavy and practically unreadable.

4

Yes, that’s with the standard log level.

Any messages on the other machine’s logs?

5

Provide screenshots from both sides please.

6

Here are the screenshots from both sides: The host Naix, is giving the problem.

Syncthing_host_sandking.png1217×723 65.3 KB
Syncthing_host_naix.png1230×726 66.9 KB

7

Audrius Butkevicius, I hope you were asking for screenshots of the GUI and not of the log or something else.

8

How do you know that it’s not making any progress?

Can you expand one of the folders and one of the devices and provide a screenshot?

Also, as you expand the folder, there should be an out of sync link which shows you progress of individual files.

9

Well, it’s been in this state most of the day. I forgot to mention that the devices are on the same physical wireless lan. I normally don’t have to wait for more than 30 minutes for a sync to complete. In this case things are most definitely broken for me because even the folder with 18 tiny files in it has been stuck at 1% for the entire day.

I noticed that the devices were connecting to each other over IPv6. Is the use of IPv6 a new change. Is it worth trying to force it back to IPv4?

Syncthing_host_naix2.png1218×819 93.2 KB

10

I’ve been dealing with the same problem after upgrading to 0.13.x:

Stuck here:

No data is being transferred:

Pasted image885×166 15.6 KB

The following is all that is logged:

[QE3BW] 2016/06/03 10:35:54 rwfolder.go:279: INFO: Folder "smswest-stagedstoreman" isn't making progress. Pausing puller for 1m0s.
[QE3BW] 2016/06/03 10:38:10 rwfolder.go:279: INFO: Folder "smswest-stagedstoreman" isn't making progress. Pausing puller for 1m0s.
[QE3BW] 2016/06/03 10:40:29 rwfolder.go:279: INFO: Folder "smswest-stagedstoreman" isn't making progress. Pausing puller for 1m0s.

No progress was made for days. Restarting Syncthing on both ends. Upgrading to latest version. Pausing, resuming. Nothing helped.

For two folders I was able to get data syncing again by removing and re-adding the folder in the Syncthing config.

I should note that all of the folders that would not sync are the receiver for a Master folder on the other end.

I have one more folder that does not sync. If you would like to have me try something or gather some data please let me know.

Server with Master folders: Windows Server 2012 R2 x64. Syncthing 0.13.5 Server with on-master folder: Debian Linux/Wheezy x64. Syncthing 0.13.5 Devices are connected by a 50M WAN.

11

So can you post the full log?

12

https://nopaste.me/view/5c00f18c

13

Does/Did it work with any previous version? Also, you seem to be syncing databases, that simply won’t work, as they usually use memory mapped files.

14

It worked up to v13. When I upgraded to v13 a few days ago the problem began. I am syncing dumps of databases that are made nighty, not live database files.

Another symptom is that there is a spinner next to “Failed Items”.

15

I get the continuous spinner as well. I’m downgrading to 12.x at the moment to see if that works without issue.

16

I downgraded to 0.12.23 on the suspect machine and 0.12.24 on the other machine. I’m using packages from the package cache so I use the nearest version I can find.

Anyway syncing works for this version. It took very long because it had to scan all the directories from scratch but it definitely works. I haven’t been able to use syncthing-inotify when using older packages (It’s a package management nightmare).

I still get the continuous spinner, for some failed items though. But overall I can confirm that 0.12.x still works for me.

17

So I don’t have a good idea how to debug this remotely, given there is no meaningful error message. Best suggestion I can make is either provide access to the environment with the issue for me to debug, or get exact steps for me to repro locally.

18

Thanks Audrius

I’ve seen that a fresh environment works fine with 0.13 That is if I nuke my config file and add new settings, then it works better than I’ve seen in a long time. I synced about 3GB or data in under 30 min (Didn’t do precise measurements)

So either my data or the config is causing the breakage. I suspect the latter. I’ll narrow it down until I have something concrete.

19

It seems somehow during my experiments, I’ve managed to get it into a permanent working state. I guess it’s a good thing, but a bit frustrating that I couldn’t figure out the cause.

Thanks everyone for your comments. I post more info if I get it into the broken state again (but hopefully not).

20

I’m experiencing the same problem… Syncthing works fine for new files but somhow some files got out of sync on 6 folders and it keeps using CPU resources as crazy even when forced to use only one core and pausing for 1 min when no progress can be made sequentially for the out of sync folders. Would be nice to have a direct way of ‘resetting’ a folder and of changing the pausing interval and or specifically ignoring files… also getting infinite wheel turning for the failed items, so I don’t even know which files are the offending ones (not that would help much as we are talking about hundreds of thousands of files and abour 250Gb of out of sync data)