I’m running Syncthing on three compouters: A (Windows), B (Windows), and C (FreeBSD). I have a few folders that are shared across all three machines. The folder appears to be in sync across all three machines (manually computed checksums for all files check out on all machines). However, the status on each machine is as follows:
A: All folders up-to-date, device B up-to-date, device C Syncing (78%)
B: All folders up-to-date, device A up-to-date, device C Syncing (78%)
C: All folders up-to-date, device A up-to-date, device B up-to-date
Strangely, devices A and B think C is out-of-sync, but device C thinks it’s up-to-date.
From the device A or B GUI, I click on Out of Sync Items: 17,595 items, and I don’t see any errors – just a list of files which are reported as out-of-sync, but in fact appear to be up-to-date.
This has persisted for days. I have tried shutting down syncthing on all machines and restarting, but result is the same. All machines running v1.17.0.
In my case it’s mixed up now. It is still described as in my first post above. Now I have specified files in the status window of one device which are deleted and no longer exist. It is not clear, where this information comes from. Also a --reset-deltas can not solve this problem.
I can’t say exactly when the problem begun, since there are already version jumps between. However, I think that with v1.16.0 this state has arise itself at some point and cannot be remedied.
Which folder(s) are the out-of-sync items from? Does their ID and local/global states match (expanded folder in the screenshot would expose that info)? If the answer is yes, it’s most likely the mysterious missing indexes problem. You could confirm with syncthing cli debug <folder-id> <file-path>, however that wouldn’t help know how the problem happened in the first place. The procedure to get rid of this problem (if it is indeed the problem), is to syncthing once with --reset-deltas on the devices that have a remote device shown as syncing (A and B in the screenshots).
You could confirm with syncthing cli debug <folder-id> <file-path>
Can you please confirm that this is the correct command? I tried running syncthing cli debug default <path to one of the out of sync files>, but I get the error No help topic for 'default'. Here, default is the Folder ID for the folder named Default Folder.
Can you also please advise on what I should be looking for in the output of this command?
however that wouldn’t help know how the problem happened in the first place
If you can think of of any methods for diagnosing the root cause, I’d be happy to try to help.
Just as a follow-up on mine: (all synced, just FYI for those that may read this later)
Not sure why mine was off on it’s deltas for one node, since it had just been added to the share a week or two ago.
Very underpowered Synology NAS, original setup I attemped to help it get caught up to the other nodes by copying the 600GB of about 150,000 files to it first.
I believe computing all the file HASH’s were too much for the tiny ARM cpu. After 6+ months it was still not done syncing. it hit a wall and was only making tiny progress.
Decided it was best to let other devives compute the HASH’s and then the NAS to just pull it from the other devices. So NAS was removed from the share, share folder removed from syncthing on the NAS, and the storage folder on the NAS manually emptied.
Syncthing share folder on the NAS was redone from scratch. Doing it this way, the tiny NAS was able to complete the Sync in less than a week. Very slow compared to other devices, but a vast improvement to this NAS.
It was at this point that the NAS thought one of the PC’s was not caught up, even though all other devices said every device was all caught up.
To correct the delta’s on the NAS, I edited the folder share on the NAS to remove the PC that it thought was wrong. Hit save and close on edit. Went right back into folder edit and re-added the PC and saved.
It started scanning the folder, and within a couple hours it showed all as synced correctly.
Thanks. I’d need that info from both devices to compare them, i.e. if in the web UI of A, C is shown out-of-sync, I need it from both A and C.
If you can reproduce it, I can try to follow your steps or if that doesn’t work, you can enable debug logging which hopefully contains clues. However likely it “just happens”, in which case I don’t know anything to diagnose it.
Thanks. As suspected it is the missing index issue: Everything is the same and looks good, except that device A doesn’t have the info from one remote device (one entry less in “availability”. There’s some chance v1.18.0 will fix the cause. To know what the cause really is, the problem needs to be reproducible.
Ah and you don’t need to hide the IDs in the version, they are equivalent to what you left not hidden in the availability part. Also the version counter is not sensitive info. However both is usually relevant in debugging (here it’s clear/likely from the left-over bits that everything is the same and that’s all I needed to know).
Thanks Simon! Now that we’ve confirmed that it’s indeed the missing index issue, I went ahead and ran syncthing --reset-deltas which appears to have cleared up the issue for now. I’ll continue to monitor to see if the issue pops up again.
Ah and you don’t need to hide the IDs in the version
Thanks for the heads up. For future reference, is there any potentially sensitive information in the debug output besides the file name?
The full device IDs are “semi-sensitive”: They don’t give anyone access to your stuff, but they allow them to try to connect to you (get “device wants to connect” prompts) which would be annoying. Other than that, no.