Unable to solve "Non-increasing sequence detected" error

[SDIT (Sergio Davies)]

Hello everyone!

I have been introduced to Syncthing through a friend and decided to give it a go (so I’m relatively new to this software).

I have two endpoints (Linux PCs) each with two instances of Syncthing running under different users: a workstation and a NAS.

The system was performing the initial synchronisation of two folders (approx 200GB each), which completed this morning, so I went to check that everything was ok.

To my surprise I found that not all the files were synchronised between the two endpoints: even though the local and remote file counts were showing the same numbers (no ignore pattern list set), some files were present only on the source endpoint, but not on the destination endpoint.

At this point I tried to figure out if there was a problem in the scan, performing multiple “rescans” of the folders on both endpoints, to no effect.

I decided to invalidate the databases on both endpoints (/usr/bin/syncthing -no-browser -no-restart -logflags=0 -reset-database), and let Syncthing recreate them. At this point I started seeing this error:

“Non-increasing sequence detected: Checking and repairing the db…”

This is an error which I seem unable to get rid of: I even deleted the “index” folders in the configuration folder, but again, to no effect. From the log, this error shows up approx. 35 seconds after service startup, and I believe it to be connected with one particular folder which was being synchronised between the two endpoints.

The log shows:

2020-05-17 16:54:22 Non-increasing sequence detected: Checking and repairing the db...
2020-05-17 16:54:22 Repaired %v sequence entries in database 690
2020-05-17 16:54:52 Non-increasing sequence detected: Checking and repairing the db...
2020-05-17 16:54:52 Repaired %v sequence entries in database 24
2020-05-17 16:59:37 Non-increasing sequence detected: Checking and repairing the db...
2020-05-17 16:59:37 Repaired %v sequence entries in database 92

I am not sure what else I could try to recover from this error, apart from deleting the entire Syncthing, including configuration and datafiles, and reinstalling it on both machines (and hoping for the best), but this would require a long time to complete, and I’d rather help to improve the system.

Does anyone have suggestions how to address it? Thank you!

[imsodin (Simon)]

What version are you on?

Just as a general remark: Don’t consider resetting the database a routine action - it is not. It is a severe intervention and if done wrong, may cause other problems. And even if done right, might just be overkill.

[SDIT (Sergio Davies)]

Hi, Thank you for your quick answer!

Yes, I know it is a severe operation, and in fact I did it after trying everything else I could think of.

I am currently on version 1.5.0 on arch linux installed from the official repository: https://www.archlinux.org/packages/community/x86_64/syncthing/

The installation procedure is: https://wiki.archlinux.org/index.php/Syncthing

I instantiated two daemons one per user, moving the communication port of one of the daemons to 22001 and GUI on 8385.

Thank you for any suggestion!

[imsodin (Simon)]

@SDIT

To my surprise I found that not all the files were synchronised between the two endpoints: even though the local and remote file counts were showing the same numbers (no ignore pattern list set), some files were present only on the source endpoint, but not on the destination endpoint.

As in they weren’t the same on disk or there were indications in Syncthing’s web UI that they weren’t synced? In either case if it happens again, please report back explaining the situation (screenshots help) so we can investigate further.

@SDIT

“Non-increasing sequence detected: Checking and repairing the db…”

This is an error which I seem unable to get rid of: I even deleted the “index” folders in the configuration folder, but again, to no effect. From the log, this error shows up approx. 35 seconds after service startup, and I believe it to be connected with one particular folder which was being synchronised between the two endpoints.

For v1.5.0 there’s no known cause for this behaviour (to me). And as starting again from scratch (deleting the database while syncthing is not running on all devices at the same time) didn’t help, there’s no simple solution and this requires debugging:
As the error message suggests, the error should be resolved automatically. If this pops up time and time again, it either isn’t correctly resolved or happens again and again. In any case, please enable db debug logging (either when starting Syncthing with the STTRACE env var or in the UI in actions>logs) and provide those logs here.

[SDIT (Sergio Davies)]

@imsodin

@SDIT

To my surprise I found that not all the files were synchronised between the two endpoints: even though the local and remote file counts were showing the same numbers (no ignore pattern list set), some files were present only on the source endpoint, but not on the destination endpoint.

As in they weren’t the same on disk or there were indications in Syncthing’s web UI that they weren’t synced? In either case if it happens again, please report back explaining the situation (screenshots help) so we can investigate further.

There were files on the source endpoint which were not present on the destination endpoint. The two endpoints are two different PCs, with two separate sets of hard drives. However, the graphical interface said that the file and folder counts on the source and the destination endpoints were equal.

@imsodin

For v1.5.0 there’s no known cause for this behaviour (to me). And as starting again from scratch (deleting the database while syncthing is not running on all devices at the same time) didn’t help, there’s no simple solution and this requires debugging: As the error message suggests, the error should be resolved automatically. If this pops up time and time again, it either isn’t correctly resolved or happens again and again. In any case, please enable db debug logging (either when starting Syncthing with the STTRACE env var or in the UI in actions>logs) and provide those logs here.

Ok. I tried to run this again, and something happened: syncthing crashed (with a panic file) and a Warning:

[D3OVX] WARNING: Sequence index corruption (folder (deleted), file (deleted)): sequence 4843 != expected 4005

The output of this experiment is rather huge (210.5MB), and contains filenames and folder, which I’d rather not show here. Can I send you a link to download both the files via email?

Thank you always for the quick reply!

[imsodin (Simon)]

That panic is a side-effect of enabling db debug logging (that should have changed a while ago but I missed it). Anyway: Easiest is if you write me a direct message here in the forum with a link to that log - thanks.

[terry]

Sergio, Click on Simons name then you get the option to send a private message

[SDIT (Sergio Davies)]

Thank you Terry,

For some reasons I’m unable to do that:

[imsodin (Simon)]

I just messaged you privately, you can remove the files - thanks. And I upped your trust level, maybe that was the reason you couldn’t send direct messages (wasn’t aware of any such limit).

The problem you are having is fixed in v1.6.0-rc.4 - it came up in a different context: The same file is update multiple times in the db in a single batch of updates - which Syncthing doesn’t handle well.

However the underlying cause there (a new feature added in v1.6.0) can’t be the underlying cause for you, given you are on v1.5.0. If you do have the time, it would be very helpful if we could determine what is causing it for you. For that you’d have to do the same thing again, but this time with all of model,scanner,db flags enabled (this will make the logs even much bigger). And however you start it seems to suppress timestamps and file names, which makes it much harder to filter. Please run the Syncthing binary directly (and/or somehow add -logflags 23 as argument) to enable. If it’s still manageable to put online, please do, otherwise I’ll ask you for some grep excerpts to get me started.

[SDIT (Sergio Davies)]

Thank you Simon!

I just did this, and the log Is now uploaded. I just sent you the link privately.

I’m not familiar with your code, but I would dare to suggest a theory: what if there is a race condition in the portion of the code listing the directories or hashing the files?

Thank you for your help!

[SDIT (Sergio Davies)]

For future reference: the outcome of this analysis so far is:

@SDIT

I have tried a different setup: different PC, windows 10 OS, and I tried to share a folder which contained a Google drive mount point. Syncthing generated the exact same error (“Non increasing sequence detected”), even though the software I use under windows to mount this file system is different (not GCSF but the official “Google sync”).

This happened even if the mount point was in the ignore list.

So I would say, now, that it is safe to assume that having a Google Drive mounted as a subfolder within the Syncthing shared folder generates problem with the thread that walks through the file system.

Of course, when I removed the Google Drive mountpoint from within the Syncthing shared folder, everything started working fine.