a family member of mine was running into an issue with syncthing today. I had a closer inspection of the issue and it appears to be potentially closely related to recent threads:
Sadly, I don’t have any screenshots or log material at this time, as it wasn’t on any computer I have regular access to. I do however have a theory on what might have happened.
The timeline of events, as I can reconstruct it, was:
Syncthing was installed on a fresh (Windows) computer.
This new syncthing instance was connected to a NAS, also running syncthing, that had a bunch of folders available.
From the NAS, a few folders where shared with the new device. They were accepted on the new device, folder types should have been send-receive both ends. No ignore patterns anywhere.
Initial index exchange for the folder started, and folders eventually started syncthing. Device was left unattended overnight for syncing.
The next morning, the new device claimed to be up to date on all folders. It was still connected to the NAS.
However, lots of files were missing on a single syncthing folder. That folder should have contained around 40.000 files, ~150 GB total, but only had ~11.000 files.
Looking at syncthings local and global state revealed the following:
NAS:
Global State: ~40.000 files, ~150 GB
Local State: ~40.000 files, ~150 GB (exact match)
New device:
Global State: ~11.000 files, ~20 GB
Local State: ~11.000 files, ~20 GB (exact match)
This was an obvious discrepancy - the new syncthing device had a completly mismatching global state and thus wasn’t syncing almost 29.000 files. Restarting both the new device as well as the NAS did not change anything.
I eventually ran syncthing -reset-deltas on the new device.
Global state immediatly started updating (increasing) and soon reached ~40.000 files, ~150 GB, matching the NAS.
Syncing resumed and is probably still running as of now.
So it seemed that the new device had some issues with the initial index exchange and did not get the full dataset?
Based on the fact that there seem to be multiple recent threads that report something similar I wonder if there’s a bug here somewhere. Both of the threads named above seemed to be having issues with new devices being added to a cluster and then only performing a partial sync, maybe due to index exchange not running correctly.
Again, sorry I don’t think I have logs on this one. Probably also not easy to reproduce either. Both devices were on 1.20.2
I’ve faced similar issues, quite a few times, but unfortunately only in a real environment with a dozen devices or so, so it was always impossible to trace what the culprit could be. The problem seemed to only affect the very first sync. I then always “fixed” it by removing and re-adding the problematic folders.
Of course, I’ve also been unable to reproduce the issue in my test environment, so I can’t provide any logs or more helpful information .
I think it’s clear there’s a bug somewhere, indeed. It’s never happened to myself and I haven’t been able to find steps that reproduce it, just.
It’s funny that it seems to happen on “initial” setup, with the solution variously being --reset-deltas and removing/re-adding folders, both of which are ways of getting back to the initial sync that caused the issue initially…
I have a feeling that the full index exchange sometimes work and sometimes it doesn’t, so it won’t work the first time but it does the second. No factual data to support this claim but it at least matches the observations.
I think the clue here may be that the first data exchange usually takes time (to transfer the files) and there can be connection drops/stalls in the process. After doing the reset, the data transfer is usually only about the indexes, as the files themselves are already there, so everything takes very little time.
Just out of curiosity; does the transfer of the actual files start before the entire index is transferred to the new node?
Like if the index is transferred in chunks, and transfer of the files in the chunk start immediately?
So the index transfer might be slowed down by the file transfer. But it should still complete eventually, unless there is some kind of timeout mechanism involved?
They run independently, no timeouts - index sending is a never ending process, more changes can always happen. Of course if there are no indexes, nothing is sent, but the sender is still up and running and waiting for new index changes to be sent. And generally index just being metadata it will send the initial bunch of indexes significantly faster than syncing the initial data.
.