Synking stuck peermanently

Hello all!

So the syncthing instance on my NAS has totally stopped syncing with everything:

Dont really know why, as you can see it finds every host that is online (the single disconnected client is a ATM powered off laptop) and actually sees the changes but no progress is being made. The “server” (main node that everything syncs to) is a truenas scale container, version 1.27.2 . Three linux client from witch 2 is on 1.18.0-ds1 and one 1.19.2-ds1 (syncthing installed from distro repos). Plus an android phone with the official app matching the servers version.

Any ideas where should i start?

What do the Syncthing folders that are stuck say when you expand the folder to show the details?

Nothing unusual (minus the “out of sync items”):

Those out-of-sync items are what’s holding up the sync progress (it’s 3 of the 4 files – 75% – that need to be synced).

Click on the “3 items, ~409 KiB” hyperlink to see what Syncthing says about them.

No errors, just color marked as downloading:

hmmm… so one file ujev3.jpg is holding things up. Since that folder is shared with multiple devices, you’ll need to track down the device sending the file (Syncthing’s log might help: Actions → Logs).

Realistically speaking it is only shared by two device with the server serving as a go-between. Found a few entries:

2024-01-21 20:06:24 Puller (folder "Share-SC" (<id>), item "ujev2.jpg"): syncing: no connected device has the required version of this file
2024-01-21 20:06:24 Puller (folder "Share-SC" (<id>), item "ujev3.jpg"): syncing: no connected device has the required version of this file
2024-01-21 20:06:24 Puller (folder "Share-SC" (<id>), item "újév.jpg"): syncing: no connected device has the required version of this file
2024-01-21 20:06:24 "Share-SC" (<id>): Failed to sync 3 items

IDK whats up with not having the correct version. Only one device has them (the blurred out one) so there shouldnt be any conflicts or hickups.

2024-01-22 00:05:03 Established secure connection to 2KGVW2X at 172.16.0.90:22000-10.125.210.180:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5TCGA6NQ15A3ALMC74G3OLNJOK
2024-01-22 00:05:03 Device 2KGVW2X client is "syncthing v1.27.2" named "FP4" at 172.16.0.90:22000-10.125.210.180:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5TCGA6NQ15A3ALMC74G3OLNJOK
2024-01-22 00:06:54 Lost primary connection to 2KGVW2X at 172.16.0.90:22000-10.125.210.180:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5TCGA6NQ15A3ALMC74G3OLNJOK: reading message: read tcp 172.16.0.90:22000->10.125.210.180:22000: read: connection reset by peer (0 remain)
2024-01-22 00:06:54 Connection to 2KGVW2X at 172.16.0.90:22000-10.125.210.180:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5TCGA6NQ15A3ALMC74G3OLNJOK closed: reading message: read tcp 172.16.0.90:22000->10.125.210.180:22000: read: connection reset by peer

Hmmmm… :thinking:

(172.16.0.90 is the internal network for k3s on the truenas host, 10.125.210.180 is the device stting on the same lan as the NAS)

/edit –reset-deltas and --reset-databse doesnt fix this.

Unless it’s happening all the time, it’s not a problem. It’s not unusual to see things like that, especially if any of your devices are on Wi-Fi.

I was going to suggest trying that…

Do újév.jpg, ujev2.jpg and ujev3.jpg actually exist on the (redacted) device?

If not, try a “touch ujev3.jpg” on the sending device for that Syncthing folder (same for the other two files). Or if they already exist, temporarily rename them, then do the “touch” command on “zenifer”. Either way there will then be something to be synced.

Happens every 5 minutes or so, its the phone so wifi might be the culprit.

On the blurred out one? Yes. Or if you mean the one where i deleted the ID (the “server”) that one does not have the files, only hidden temporary files:

drwxr-xr-x 2 983 983      2 Apr 13  2023 .stfolder
-rw-r--r-- 1 983 983 118693 Jan 22 05:06 .syncthing.ujev2.jpg.tmp
-rw-rw-r-- 1 983 983 133078 Jan 22 05:06 .syncthing.ujev3.jpg.tmp
-rw-r--r-- 1 983 983 233276 Jan 22 05:06 .syncthing.újév.jpg.tmp
-rw-rw-r-- 1 983 983  65043 Jan  1 11:23 ujev4.jpg

/edit Just caught an error on the webui of the server, it stated that the host that has the images is disconnected. Expanded it and i seen this text in red: “i/o timeout (04:58:22)”.

Took the plunge and wiped the config completely (but kept the data for now) along with the container the re-created it. After the initial config (setting ports and locking down the gui with login) i added this new container to the host list of the instance running on my desktop and shared a single folder with it. The container shortly after notified me if i want to add a new device, then about the shared folder. Accepted the device and added the folder (entered a partially existing path so the data will end up where it should but at the same time it wont have pre-existing data interfere with the sync process.

All for nothing, the instance running in the container is sitting pretty thinking its all up-to-date while the desktop instance clearly sees it is not and wants to sync… :face_with_symbols_over_mouth:

/update Added syncthing to apt sources and now both instances are on the same version. No change in behavior.

If the devices are not connecting, then what you observe is normal. The one side knows that there is data it has never synced with the other side, thus it shows the remote device out of sync. The other side doesn’t know about the extra data yet, so it concluded that the existing data is consistent with the last known global state. In order to make it notice there’s a difference, it needs a connection to the first device again and update its global state to include that remote’s changes.

You need to solve any prevalent connection problems before asking about synchronization state.

So do they ever connect successfully? What other errors are being shown for the addresses tried on each side?

Oh i wish if it were that simple: https://content.section9.me/im/scf/sc4.png

(Since zenifer says up-to-date for the other host i assume it is connected.)

Logs:

https://content.section9.me/im/scf/dell-gpc-linux.log

https://content.section9.me/im/scf/zenifer.log

Can you expand the folders (click on them) in the screenshots?

I could but now it is synced, last thing i did was to enable the “Host Network” for the container but even then it wouldnt start to sync. At least as long as i kept an eye on it then closed all tabs and went to sleep. Havent took a look at it until now… :thinking:

Ill try to add more folders.

/EDIT Looks like it works now, guess ill have to take it up with the folks making the container in TN Scale.

Spoke too soon maybe? Added back phone and same mess is happening again… sigh Guess im still not done poking around.