Syncthing destroying data

Support
1

Hi, I’m new to syncthing, yet very concerned on what I’ve experienced.

Syncthing is configured as follows:

  • Running from docker compose (via portainer) with “network_mode: host” set as to eliminate any docker/host NAT issues
  • Sync Protocol Listen Addresses at each device: tcp6://:22000
  • All Discovery, NAT & Relay disabled
  • Each device’s address specified as: tcp6://deviceX.domain.com:22000

In all regards, a direct connection between two devices.

On adding a folder and then accepting it a the destination device, files from device 1 (i.e. DSC0001.JPG) is created at device 2, yet only as tmp files (i.e. .syncthing.DSC0001.JPG.tmp)

The sync process simply doesn’t complete.

At the source device, the GUI states “Syncing (0%, 1.29 MiB)”, also reflecting the file as being out of sync.

At the destination device the GUI states “Syncing (14%, 1.1 Mib)” also reflecting the file as being out of sync.

… hours go by without any change …

Restarting, pausing et all other attempts has no effect.

And then the following shocker :astonished:

On scp’ing an image file to the source device’s folder, the filename is created, yet no content is transferred. On terminating the copy process, syncthing creates the file at the destination device, empty, and then syncs it back to the source device.

Now the question would be. Is it just a config issue on my side that can cause real damage or should there be a big warning put out to newcomers?

2

I have no idea what going on here, but gut feeling says there’s something very unusual or flaky about your storage? There should be complaints from Syncthing in the logs, what do those say?

3

Only the following INFORMATIVE, which ain’t helping:

“source” device:

INFO: "Test" (dn6hj-uhcqd): Failed to sync 2 items
INFO: Folder "Test" (dn6hj-uhcqd) isn't making sync progress - retrying in 17m3s.

“destination” device:

INFO: Puller (folder "Test" (dn6hj-uhcqd), item "s1.txt"): syncing: no connected device has the required version of this file
INFO: Puller (folder "Test" (dn6hj-uhcqd), item "s2.bmp"): syncing: no connected device has the required version of this file
INFO: "Test" (dn6hj-uhcqd): Failed to sync 2 items
INFO: Folder "Test" (dn6hj-uhcqd) isn't making sync progress - retrying in 8m20s.
4

Help me get this clear: you’re scping a file to Device A from Device B, and scp doesn’t work properly: it creates an empty file and hangs there without transferring the actual data? Something must be horribly wrong with your system.

Now, after you’ve terminated scp, you end up with the empty file on Device A that is newer than the file with the same filename than the one on Device B. Syncthing has no way of knowing how the new empty file got there. Syncthing has no way of knowing that it was an scp attempt that went haywire, all Syncthing knows is that Device A has a fresher version of file than Device B, and that these versions differ, so Synching does what it’s designed to do: replace the version on Device B with the new empty one from Device A.

So basically, something with Device A is very wrong (presumably the storage that allows to create files but not to write actual data into them). Syncthing fails to sync files onto that device, but does it in a graceful way and without destroying any data on other devices. And only when you confuse Syncthing by using alternative means of file transferring (and failing at that due to faulty system) is Syncthing actually damaging the data. Am I correct?

5

Device A is presently serving content and has so over the past 3 years. Which is why Syncthing is coming into play. (realtime fallback option for the day it does go south)

I believed the SCP example only confirmed what I understood could relate to Syncthing. Be it a network issue, as there weren’t any helpful info in logs I were tailing.

Since I was contemplated more in the direction of issues within the virtualisation fabric, I’ve flashed the NUCs with the latest firmware. Updated each hypervisor and recreated all containers with the latest images.

After the upgrades, I can still read and write files as always. Media streaming works flawlessly. Yet upon starting Syncthing at both devices, it’s still not syncing the smallest of test files. It was at this point previously that I started to pause & restart processes which resulted in noticing the scp situation.

At present I can confirm that scp’ing from either to another is working flawlessly. Only Syncthing isn’t syncing the smallest of sample files.

I’ve now thrown a 174GB & 159GB folder at Syncting, and nothing has happened as yet, except for Syncthing still fixating on the Test folder…

At Device A:

[ELERZ] 06:18:26 INFO: Device BH63AO3-DWON7QJ client is "syncthing v1.9.0" named "B.domain.com" at [DEVICE_A]:22000-[DEVICE_B]:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
[ELERZ] 06:26:16 INFO: Established secure connection to BH63AO3-DWON7QJ at [DEVICE_A]:22000-[DEVICE_B]:57380/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
[ELERZ] 06:26:16 INFO: Replacing old connection [DEVICE_A]:22000-[DEVICE_B]:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256 with [DEVICE_A]:22000-[DEVICE_B]:57380/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256 for BH63AO3-DWON7QJ
[ELERZ] 06:26:26 INFO: Connection to BH63AO3-DWON7QJ at [DEVICE_A]:22000-[DEVICE_B]:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256 closed: writing message: write tcp6 [DEVICE_A]:22000->[DEVICE_B]:22000: use of closed network connection
[ELERZ] 06:26:26 INFO: Device BH63AO3-DWON7QJ client is "syncthing v1.9.0" named "B.domain.com" at [DEVICE_A]:22000-[DEVICE_B]:57380/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
[ELERZ] 06:34:16 INFO: Established secure connection to BH63AO3-DWON7QJ at [DEVICE_A]:22000-[DEVICE_B]:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
[ELERZ] 06:34:16 INFO: Replacing old connection [DEVICE_A]:22000-[DEVICE_B]:57380/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256 with [DEVICE_A]:22000-[DEVICE_B]:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256 for BH63AO3-DWON7QJ
[ELERZ] 06:34:26 INFO: Connection to BH63AO3-DWON7QJ at [DEVICE_A]:22000-[DEVICE_B]:57380/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256 closed: replacing connection
[ELERZ] 06:34:26 INFO: Device BH63AO3-DWON7QJ client is "syncthing v1.9.0" named "B.domain.com" at [DEVICE_A]:22000-[DEVICE_B]:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256

At Device B:

[BH63A] 06:26:16 INFO: Device ELERZ6H-N6AVEQZ client is "syncthing v1.9.0" named "A.domain.com" at [DEVICE_B]:57380-[DEVICE_A]:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[BH63A] 06:33:46 INFO: Connection to ELERZ6H-N6AVEQZ at [DEVICE_B]:57380-[DEVICE_A]:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256 closed: read timeout
[BH63A] 06:33:46 INFO: Puller (folder "Test" (dn6hj-uhcqd), item "testA.txt"): syncing: no connected device has the required version of this file
[BH63A] 06:33:46 INFO: Puller (folder "Test" (dn6hj-uhcqd), item "DeviceA.txt"): syncing: chmod /vault/Test/DeviceA.txt: no such file or directory
[BH63A] 06:33:46 INFO: "Test" (dn6hj-uhcqd): Failed to sync 2 items
[BH63A] 06:33:46 INFO: Folder "Test" (dn6hj-uhcqd) isn't making sync progress - retrying in 8m20s.
[BH63A] 06:33:46 INFO: Puller (folder "Test" (dn6hj-uhcqd), item "DeviceA.txt"): syncing: chmod /vault/Test/DeviceA.txt: no such file or directory
[BH63A] 06:33:46 INFO: Puller (folder "Test" (dn6hj-uhcqd), item "testA.txt"): syncing: no connected device has the required version of this file
[BH63A] 06:33:46 INFO: "Test" (dn6hj-uhcqd): Failed to sync 2 items
[BH63A] 06:33:46 INFO: Folder "Test" (dn6hj-uhcqd) isn't making sync progress - retrying in 1m0s.
[BH63A] 06:33:46 INFO: Puller (folder "Test" (dn6hj-uhcqd), item "DeviceA.txt"): syncing: no connected device has the required version of this file
[BH63A] 06:33:46 INFO: Puller (folder "Test" (dn6hj-uhcqd), item "testA.txt"): syncing: no connected device has the required version of this file
[BH63A] 06:33:46 INFO: "Test" (dn6hj-uhcqd): Failed to sync 2 items
[BH63A] 06:33:46 INFO: Folder "Test" (dn6hj-uhcqd) isn't making sync progress - retrying in 1m0s.
[BH63A] 06:34:16 INFO: Established secure connection to ELERZ6H-N6AVEQZ at [DEVICE_B]:22000-[DEVICE_A]:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[BH63A] 06:34:16 INFO: Device ELERZ6H-N6AVEQZ client is "syncthing v1.9.0" named "A.domain.com" at [DEVICE_B]:22000-[DEVICE_A]:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256

After an all nighter, I’m stumped …

6

In the log excerpt only one side mentions anything about folders and the pull error is most likely due to the timed out connection. To get an idea of what your setup and thus the problem may be, please post screenshots from both sides.

8

I was about to give up on syncthing …

… and then by disabling “TX Checksum Offloading” at the virtual network interface that was timing out, syncthing came to life !

A hint in the syncthing log files could save the next newcomer easily 4 days of lost sleep. :wink:

9

Logs of what?

10

The attainable speed between two devices.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.