Device stays out of sync until restart

imsodin · November 20, 2020, 1:05pm

Yes Andy, there have been significant changes related to this in v1.11.1 and v1.12.0 - as documented in the changelog. Thanks to all the debugging in this thread we have actually found problems and are fixing them right now. Downgrading may be a necessary and effective measure, but it’s not a permanent solution nor does it help fix problems.

Alex · November 22, 2020, 1:45pm

Tested latest fixes on the test setup for a few days without issues so I also tried it today on the real instances: also no out of sync or other issues even after multiple reboots So i guess my problems are fixed.

imsodin · November 22, 2020, 3:49pm

Great news!

I know I sound like a broken record, but I can’t help it: Thank you so much for providing comprehensive and speedy info while debugging this issue (well, issues ).

Alex · November 22, 2020, 4:16pm

Thank you for the quick fixes

Andy · November 22, 2020, 6:16pm

Which are meant?

Alex · November 22, 2020, 7:35pm

Those should be the PRs fixing the issues found here (or even more), latest nightly has them, next release containing them will probably take a while:

github.com/syncthing/syncthing

lib/model: Never send unpaused folder without index info

syncthing:main ← imsodin:model/ccNoFset

opened 12:43PM - 20 Nov 20 UTC

imsodin

+35 -5

More fixes regarding races between folders being started and cluster configs bas…ed on the logs provided in https://forum.syncthing.net/t/device-stays-out-of-sync-until-restart/15891 There's an issue with sending cluster configs: When a folder is not paused, but hasn't been started yet, we send an info that says that the folder is running, but there's no index id/sequence. The remote then drops all info on us. Once the folder is started, we happily start sending from a non-zero starting sequence. Thus the remote will be missing index info from us between 0 and that starting sequence. The fix for this is to pretend the folder is paused if it isn't running, even if it isn't paused in config. Then when it is actually started resend the cluster-config including the proper index info. How I think this problem persists goes like this (didn't check it): 1. Devices A and B in sync, B not runnig. 2. B starts. 3. Connection happens before folder on B is started. A sends valid index info, B sends zeros (as fset is missing). 4. A drops index info about B, because of the zero index info, while B starts sending index info starting with the non-zero index received from A. For reasons we do retain the index ID of B on A when dropping index info. Result: A is missing index info between sequence 0 and sequence sent to B in step 3 above. Even if they reconnect, index IDs still match -> no index retransfer. ### Testing Added a unit test that makes sure we don't send a unpaused folder with zero index id.

github.com/syncthing/syncthing

lib/model: Retain index info for new folders/devs (ref #7100)

syncthing:main ← imsodin:model/indexFollowup7100

opened 12:43PM - 20 Nov 20 UTC

imsodin

+59 -59

More fixes regarding races between folders being started and cluster configs bei…ng processed based on the logs provided in https://forum.syncthing.net/t/device-stays-out-of-sync-until-restart/15891 The fix for race on incoming cluster-configs in #7100 works, but it's an unclean and partial fix: Index senders are started without index info, as the folder/device is new. That's correct for the starting sequence, but it's still bad because we don't store the index id. Meaning on a successive connect, we will re-exchange all index data. That is fixed by always storing the index data received in cluster-configs, even if we don't share the folder so far. And we should also resend cluster-configs when adding folders on starting the model - connections may happen before that. ### Testing Extended the unit test introduced in #7100 to cover this scenario too.

Andy · November 24, 2020, 7:51pm

Is the result maybe v1.12.0-rc.4 ?

calmh · November 24, 2020, 9:08pm

New 0.12.0 RC would be for fixing things we specifically broke in 0.12.0, I don’t think these are that.

Andy · November 24, 2020, 9:36pm

So is maybe fixed in v1.13.x? Since I think about the stuck problem with the versions after v1.10.0.

calmh · November 24, 2020, 9:39pm

The bugfixes being made currently will go into the release after 0.12.0, yes. (For the moment it looks like we’ll call it 0.12.1.) There are also the nightly builds for the impatient. (I realise these are not easily available for your Synologies.)

Andy · November 25, 2020, 7:46am

On a Windows 10 device, I tested whether the current nightlys (I’m an impatient tester … ) change something. However, it is then a mixed environment with v1.10.0 with v1.12.0-rc.3-nightly, maybe that’s not okay.

On the Windows 10 device with v1.12.0-rc.3-nightly and all the others except one everything was synchronized, but on that one device with v1.10.0 the synchronization got stuck:

syncthing_v112nightly_1

syncthing_v112nightly_2

I then went back to v1.10.0 on the Windows 10 device, after which all devices were OK and synchronized again.

Andy · December 2, 2020, 9:29am

Has the issue been resolved in the upcoming v1.12.1, or will it come later?

imsodin · December 2, 2020, 9:39am

The sentence just before the extract you quoted answers that:

Andy · December 2, 2020, 9:43am

For me it seems not sure.

calmh · December 2, 2020, 9:45am

What is not absolutely certain is precisely what the version will be called because that depends on what changes goes into the version, and development is still ongoing.

Andy · December 2, 2020, 9:54am

That’s how I understood it. It seems to me that with the beginning of v1.11.x there is a bug or condition that will continue to exist until v1.1x.x until it is corrected or changed. Such is my interpretation and hence my question. Maybe there is an assessment of it.

calmh · December 2, 2020, 9:58am

Let me try again.

The bug has been solved.
It will be in the next release.
The next release will probably be called 1.12.1, but could theoretically become 1.13.0 due to changes between now and then.

system · January 1, 2021, 9:58am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.