Remote device stuck "Syncing"

[nick]

Hi,

I’m running Syncthing on three compouters: A (Windows), B (Windows), and C (FreeBSD). I have a few folders that are shared across all three machines. The folder appears to be in sync across all three machines (manually computed checksums for all files check out on all machines). However, the status on each machine is as follows:

A: All folders up-to-date, device B up-to-date, device C Syncing (78%)

B: All folders up-to-date, device A up-to-date, device C Syncing (78%)

C: All folders up-to-date, device A up-to-date, device B up-to-date

Strangely, devices A and B think C is out-of-sync, but device C thinks it’s up-to-date.

From the device A or B GUI, I click on Out of Sync Items: 17,595 items, and I don’t see any errors – just a list of files which are reported as out-of-sync, but in fact appear to be up-to-date.

This has persisted for days. I have tried shutting down syncthing on all machines and restarting, but result is the same. All machines running v1.17.0.

Any help would be appreciated!

[Chris.Karst (Christopher Karst)]

I have a similar problem:

I have 8 devices sharing a folder. 5 windows 10, 3 NAS.

All devices show themself as fully synced.

One NAS shows one Windows device as syncing 71% (out of sync files shown as being from various devices)

[galilley]

It seems very similar to these two topics:

Folders “Up to Date” but stuck in “Syncing” with empty “Out of Sync Items” v2

It’s the same effect as in the first thread (without “v2”). On one of my Synology servers I have the effect that four active devices give the same image: [grafik] The situation is the same for every device with the empty window: [grafik] I don’t have an emergency now, the synchronization works anyway. Still, I can’t get rid of the error with --reset-deltas. I will probably have to use --reset-database if another solutions doesnt work.

List of out of sync items is empty

Another case that appears in the new version of Syncthing (1.17.0). A lot of nodes share different parts of one folder selected by ignore patterns. At least two of nodes (A and B) stuck in Syncing state but shared folder is Up to Date. When I’d like to see the list of items it shows nothing. Device A: [изображение] [изображение] Device B: [B_main] [B_osi]

[Chris.Karst (Christopher Karst)]

Fyi mine does not show empty.

It says there are over 80,000 files out of sync. It shows the list of files. Over 8,000 pages worth.

[galilley]

In this case it could be similar to this one (see the last two or three posts):

New cases of “Out of sync” problem

Dear developers, I faced with a few new cases “Out of sync” problem. I have updated almost all my nodes up to version 1.17.0 and I expected that all of them will be in “Up to date” state. However, some strange things are happening that I try to describe below. There are two nodes with a few common folders. One of them has simple ignore patterns to exclude temporary files like *.pyc, ~, etc. There are no problem before with sync state (version 1.15.1). I’ve updated them one by one with a long …

But I have a quite complex ignore patterns that could make the situation worse.

[nick]

Same thing on my end – ~7000 files out-of-sync, ~700 pages worth.

I only have one pattern to ignore temporary files: ~*.xls*

[Andy (Andreas Kaelber)]

@Folders “Up to Date” but stuck in “Syncing” with empty “Out of Sync Items” v2

It’s the same effect as in the first thread (without “v2”). On one of my Synology servers I have the effect that four active devices give the same image: [grafik] The situation is the same for every device with the empty window: [grafik] I don’t have an emergency now, the synchronization works anyway. Still, I can’t get rid of the error with --reset-deltas. I will probably have to use --reset-database if another solutions doesnt work.

In my case it’s mixed up now. It is still described as in my first post above. Now I have specified files in the status window of one device which are deleted and no longer exist. It is not clear, where this information comes from. Also a --reset-deltas can not solve this problem.

I can’t say exactly when the problem begun, since there are already version jumps between. However, I think that with v1.16.0 this state has arise itself at some point and cannot be remedied.

[Chris.Karst (Christopher Karst)]

It apears as the previous replys are not related to the troubles we are having.

Can anyone offer any further assistance?

[AudriusButkevicius]

It’s always best to post screenshots to explain the problem, preferably from all devices.

[nick]

Thanks for the suggestion. Here are screenshots from each of my devices:

[imsodin (Simon)]

Which folder(s) are the out-of-sync items from? Does their ID and local/global states match (expanded folder in the screenshot would expose that info)? If the answer is yes, it’s most likely the mysterious missing indexes problem. You could confirm with syncthing cli debug <folder-id> <file-path>, however that wouldn’t help know how the problem happened in the first place. The procedure to get rid of this problem (if it is indeed the problem), is to syncthing once with --reset-deltas on the devices that have a remote device shown as syncing (A and B in the screenshots).

[nick]

Which folder(s) are the out-of-sync items from?

From the Device A GUI, I clicked under Remote Devices > Device C > Out of Sync Items and it lists Default Folder, Y, and Z.

Does their ID and local/global states match (expanded folder in the screenshot would expose that info)?

Each folder on all devices appear to have matching local/global states. Here’s an example:

You could confirm with syncthing cli debug <folder-id> <file-path>

Can you please confirm that this is the correct command? I tried running syncthing cli debug default <path to one of the out of sync files>, but I get the error No help topic for 'default'. Here, default is the Folder ID for the folder named Default Folder.

Can you also please advise on what I should be looking for in the output of this command?

however that wouldn’t help know how the problem happened in the first place

If you can think of of any methods for diagnosing the root cause, I’d be happy to try to help.

Thanks again.

[imsodin (Simon)]

@nick

Can you please confirm that this is the correct command?

nope, i forgot one bit: cli debug file

[Chris.Karst (Christopher Karst)]

Just as a follow-up on mine: (all synced, just FYI for those that may read this later)

Not sure why mine was off on it’s deltas for one node, since it had just been added to the share a week or two ago.

Brief history:

Very underpowered Synology NAS, original setup I attemped to help it get caught up to the other nodes by copying the 600GB of about 150,000 files to it first.

I believe computing all the file HASH’s were too much for the tiny ARM cpu. After 6+ months it was still not done syncing. it hit a wall and was only making tiny progress.

Decided it was best to let other devives compute the HASH’s and then the NAS to just pull it from the other devices. So NAS was removed from the share, share folder removed from syncthing on the NAS, and the storage folder on the NAS manually emptied.

Syncthing share folder on the NAS was redone from scratch. Doing it this way, the tiny NAS was able to complete the Sync in less than a week. Very slow compared to other devices, but a vast improvement to this NAS.

It was at this point that the NAS thought one of the PC’s was not caught up, even though all other devices said every device was all caught up.

To correct the delta’s on the NAS, I edited the folder share on the NAS to remove the PC that it thought was wrong. Hit save and close on edit. Went right back into folder edit and re-added the PC and saved.

It started scanning the folder, and within a couple hours it showed all as synced correctly.

[nick]

Here is the debug output for one of the “Out of Sync Items”:

[imsodin (Simon)]

@nick

Here is the debug output for one of the “Out of Sync Items”:

Thanks. I’d need that info from both devices to compare them, i.e. if in the web UI of A, C is shown out-of-sync, I need it from both A and C.

@nick

If you can think of of any methods for diagnosing the root cause, I’d be happy to try to help.

If you can reproduce it, I can try to follow your steps or if that doesn’t work, you can enable debug logging which hopefully contains clues. However likely it “just happens”, in which case I don’t know anything to diagnose it.

[nick]

Thanks. I’d need that info from both devices to compare them, i.e. if in the web UI of A, C is shown out-of-sync, I need it from both A and C.

Ah ok. The above debug output is from Device A. Here’s the corresponding output from Device C:

[imsodin (Simon)]

Thanks. As suspected it is the missing index issue: Everything is the same and looks good, except that device A doesn’t have the info from one remote device (one entry less in “availability”. There’s some chance v1.18.0 will fix the cause. To know what the cause really is, the problem needs to be reproducible.

Ah and you don’t need to hide the IDs in the version, they are equivalent to what you left not hidden in the availability part. Also the version counter is not sensitive info. However both is usually relevant in debugging (here it’s clear/likely from the left-over bits that everything is the same and that’s all I needed to know).

[nick]

Thanks Simon! Now that we’ve confirmed that it’s indeed the missing index issue, I went ahead and ran syncthing --reset-deltas which appears to have cleared up the issue for now. I’ll continue to monitor to see if the issue pops up again.

Ah and you don’t need to hide the IDs in the version

Thanks for the heads up. For future reference, is there any potentially sensitive information in the debug output besides the file name?

[imsodin (Simon)]

The full device IDs are “semi-sensitive”: They don’t give anyone access to your stuff, but they allow them to try to connect to you (get “device wants to connect” prompts) which would be annoying. Other than that, no.