I have several boxes setup with syncthing. Two main units are Synology NAS’s that receive images from remote Raspberry Pi units, all running syncthing v1.0.0.
The RPi’s sync to both Synology NAS’s. The Synology’s are almost exactly the same, using the same hardware, OS and syncthing.
The second Synology though, while receiving folder info and is preoperly connected to the remote RPi’s, is constantly stuck at 0% (on the remote RPi), whilst the Synology locally shows everything is green and doesn’t update.
The RPi’s sync fine (same folder) to the other Synology NAS. Neither end complains. There’s no permission problem. I’ve narrowed the problem to the second Synology NAS, but I don’t know where to look, or what to monitor in the logs.
Using wireshark I can see some communication between RPi’s and the Synology NAS’s, but the second unit has significant less traffic.
I’ve simplified settings, explicitly specifying addresses, disabled discovery, relaying and NAT (none of it is needed in my setup).
Yet no matter what I do, where I look, I just can’t get it to work on this one NAS. I’ve removed syncthing several times, removed folders, linked devices and relinked everything (both devices and folders are broadcasted once one end knows about a device).
Anyone have any idea what to look for?
Here’s screenshots of both ends of a Synology NAS and an RPi:
Seems the index has not been trasferred from one device to the other, either other devices are super busy or there is something wrong with the nextwork most likely.
Both are idling around 0.03% CPU (screenshots show that too). So def not busy. Network is also not congested. Both Synology NAS’s are next to each other on the same local network. Pings, traces all come back clean. Wireshark shows traffic back and forth. syncs to the first Synolgoy NAS is fine, the second (same rack, same vlan/local LAN) does not sync.
How can I check whether indexes are being transferred?
I think this is #5340 which I have a fix brewing for. In the meantime, restarting either side one with -reset-deltas is probably enough to kick it in gear.
In my case though, it doesn’t even start syncing. The receiving end (Synology NAS) doesn’t get a notification that there’s files to be synced, the sending side (Raspberry) shows there’s files ready to sync.
Audrius’s idea that the index doesn’t get transferred seems to make sense. But how do I check this? packetsniffin won’t work as syncthing encrypts traffic, so all I’ll see is traffic going back and forth, not the contents.
Can I make this visible in the logs? If so, which log option will tell me this?
Index not getting sent is precisely the issue I pointed to, with the suggested workaround to clear it up. Removing the folder on one side and re-adding it would probably also do it.
I’ve removed the folder on both sides multiple times already. Reinstalled syncthing on the Synology a couple of times too. Checked that all files are gone.
It never started syncing, even after reinstalling/recreating.
I tried -reset-deltas from the cli (invoking the syncthing binary with -reset-deltas) which then uses a default config, not the config the system normally uses (ie I use :7070 for the GUI, not the default, and when invoking syncthing from the cli it sets the default port). Both RPi and Synlogy run linux.
Would that still work or does it need to invoke syncthing using the set configuration?
Odd. The -reset-deltas thing needs to run with the same config directory, yes, or it has no effect. But removing and re-adding the folder will have the same effect. The issue I’m looking at is also random and a bit rare - the odds that it would happen multiple times for so few files are roughly zero. So probably you are suffering from something else, although I don’t know what. Everything (else) looks OK in the screenshot. It’s unusual to have compression off but it should still work.
Enabling tracing of model will show you incoming and outgoing index messages, you can enable and see it in Actions > Logs. It’s only expected that it should send the index on first connect after adding the folder, or when the files change of course.
If you touch a file on the source device, does it get sent?
At the moment, my setup is pretty reproducable. Just won’t work. I’m not entirely convinced it is syncthing that is failing, it may well be something either on the network or on the NAS. But I’ve exhausted both the network and NAS looking randomly at anything I can think of. Syncthing is the one that highlights the issue, and if it is not syncthing, it may help me to pinpoint the real cause.
I touched a file on one of the RPi’s that has nothing on it at the moment. Empty, 0 byte file. Gets synced in a few seconds. Trying the other way to sync a 30MB raw image from the NAS to the RPi just sits there though. Both devices see it but sync stays at 0%.
Touching a file on one of the RPi’s that’s stuck at 0% does nothing. The RPi doesn’t appear to rescan the folder while it’s got stuff waiting.
For a bit of history, this all used to work. Months ago I had OpenMediaVault on the RPi’s and syncthing that came with it. The Synologies were running an 0.14 version of syncthing. Over the Xmas break I reconfigured things, the Synology NAS’s OS was upgraded, and syncthing was upgraded to v1.0.0. The RPi’s now run standard Raspbian (Debian based) and syncthing v1.0.0.
Before I upgraded the RPi’s to v1.0.0 I got a problem similar to what you referred to, Sync wouldn’t finish completely.
Back to my current problem, I just noticed that the raw image I was trying to sync across, did actually make it through. There’s a 30MB tmp file on the RPi, but its not finishing. Deleting the raw file from the source (Synology), leaves the RPi in a syncing state. Once I pause the RPi, and resume, it sorts it out. Copying a file onto the RPi to let it sync to the Synology, results in the original problem. The Synology does not get the update to tell it there’s new files.
Here’s the log from the Synology NAS of the past half hour or so with ‘model’ enabled.
Tell me which options you need enabled, and I’ll get the logs. Thanks for your help BTW!
This also has the touched file sync and the NEF (raw image) I was trying to sync.
Look for DSC_0432.NEF
Also, the Synology NAS is 192.168.1.22, the RPi is 192.168.103.22
So if you remove the folder and re-add it on the other side, let things simmer for a minute, then grab the same log? That would capture the index exchange and initial attempt to get in sync.
Doing that now.
There was also no specific reason for turning off compression btw. I was just disabling anything that could potentially interfere. Same is true for permissions check, turned that off just in case.
OK, here’s logs from both ends. Interstingly the log from the RPi is a log bigger. I restarted syncthing once the folder had been removed to start a fresh log (with model enabled).
The log ends the same second they connect with the folder added. Didn’t anything happen after that point? Also, the folder is xlgro-wzrde which is not the one you had issues with in the screenshot, and it seems removed on this device rather than the other side. So I have no idea what’s going on. Sorry.
I’ve just collected new logs, I let it run since uploading the previous ones.
Argh, just noticed that the second log is not the RPi. It’s a log I previously collected. Sorry.
The attached logs are the correct ones. I deleted the folder from both ends, and then recreated it again. Which would have created a new id. I’ve used a different remote RPi as it only has one image on it. The issue is the same though.
I had a look into those logs myself as well. They didn’t contain any reference to the image that was already in the folder on the RPi. Even though the GUI showed the RPi trying to sync with the Synology and indicating the image needing to be synced. It wasn’t until I deleted the image, paused and resumed the folder, and copied a new image in the folder on the RPi that the image showed up in the log (on the RPi log, nothing shows in the Synology log).
I assume the existing image wasn’t indexed as it was already in the DB on the RPi syncthing?
Anyway, I’ve added new logs that have the attempt to sync the image.
I finally managed to narrow it down to the network. I setup an RPi on the same network as the Synology NAS and tested syncing, using exactly the same settings as my remote RPi’s. Everything worked fine.
Which really leaves only the network being the cause. Which is interesting because at every location where an RPi runs, I also have cameras which are feeding a constant stream of video to another NAS, sitting right next to the Synology NAS. The cameras feed a stead stream of about 600kbps on different ports.
I’ve also tried using different ports for syncthing (23000 and 2200) to no avail. So it must be something about the traffic that is causing issues.
Can someone elaborate on the usual/normal traffic patterns? I’d like to monitor with wireshark, but not knowing what’s normal, it’s hard to troubleshoot.
Disabling the cameras makes no difference, in most cases the network easily supports in excess of 10Mbps.
