Hi there
My Syncthing has been down for a couple of weeks and I didn’t realise. I’m seeing -
Panic at 2020-02-26T00:25:42Z panic: non-increasing sequence, current: 976 <= previous: 8696
This is 1.4.0 rc-7
Can anyone advise what I should do?
Thank you 
Hello:
Likewise, I’m seeing this on one of my machines the last couple of days.
The only change I recently made was to upgrade the SynoCommunity ST package - which would have temporarily downgraded the Syncthing binary. Since then, I’ve seen it panic with this error twice.
Running Syncthing 1.3.4 - I can provide a log privately if required.
Thanks.
So, it’s a bad sign that this happens and a log might be useful, but the check is also only performed if you have model tracing on. Turn that off for normal operations.
Thanks for the quick reply Jakob.
You’re right - I had model tracing on to debug an issue we’ve been having with deleted files re-appearing - but I’ll turn it off now.
I’ll send you the logs privately, just in case they’re of any use for you.
In 1.3.4 I’m not really surprised by this.
In 1.4.0-rc.7 it’s more surprising because that version attempts to fix this as part of the upgrade migration. Of course database problems can happen anyway due to flaky hardware or whatnot, but …
Hi there
I don’t think I have model tracing on. I’m running Syncthing via SyncTrayzor on Win10. I’ve looked in the ST interface but see nothing re tracing.
WOuld it help for me to send you a panic log direct?
@calmh Discussion already ongoing in the PR We only check for missing and duplicate sequence entries, but not that sequence keys match sequences in file infos. I made a function that checks for that and fixes it, activatable by env var. Could you have a look to see if it is sane (I did some testing…): https://github.com/syncthing/syncthing/pull/6367
@marky_uk Go to the menu at actions>logs>debug facilities and check if model is ticked. Also would you be ok to run a custom syncthing build to see if the check I wrote about above fixes the problem for you?
@calmh just told me that our existing index checker can detect this problem. Could you please download the stindex tool for your system from https://build.syncthing.net/buildConfiguration/Syncthing_Tools/55765?buildTab=artifacts (you can login as guest), stop Syncthing and then run stindex -mode idxck and report the output here - that would be very helpful!
Upsi. Low memory system with lots of data in Syncthing?
If yes, any chance you can copy the index-v0.14.0.db directory to a machine with more ram and run the test there? Or upload it somewhere so we can inspect?
Hiya
So sorry to ask. Managed to copy the files to another PC that doesn’t have syncthing installed, and downloaded the stindex file to the same folder. When running the command I’m shown - CreateFile c:\Users(path here)\index-v0.14.0.db: The system cannot find the path specified. (is another instance of Syncthing running?)
That path doesn’t exist as I don’t have Syncthing on this PC. Do I need it installed? Or create the path? Or supply params? Or something else? Thank youuu!
Hey - on the new PC, I get the same failure:
It’s got 16GB RAM, and is using about half that currently.
Forgot about that: If it’s not at the default location, you need to specify the path. So e.g. for \foo\bar\index-v0.14.0.db call stindex -mode idxck \foo\bar\index-v0.14.0.db.
You downloaded the 32bit binary (ams_386 in the screenshot), while with 16GB ram you are definitely on a 66bit system. You need to download the windows-amd64 variant.
I assumed that was on purpose on the original system, maybe it would have worked there too if it’s 64bit as well.
What a muppet I am - I’ll try with the larger file and the params!
Hiya. Running this on the original PC with 8GB RAM. How long do you think this would normally take? It’s been running since yesterday and still going!
I was wondering what’s going on, with you responding laser-fast usually
How much data do you have in Syncthing? And what does the RAM usage look like?
I just ran it locally on my production env with ~500k files and ~0.5TB data, with database on an SSD in ~10s.
Haha, I debated replying and wondered about forum etiquette - do people want to know “it’s still running” and receive boring updates, or wait until it’s done but not know if I’ve gone to Wuhan for the weekend.
Data = 1.7TB, 4.8M files. RAM usage = 5.2GB by the indexing process.
If your system starts swapping due to the index checker it’ll probably take forever.
Yeah, RAM is at 92% usage. I chose such a low spec Dell for the “server” as normally it doesn’t do much apart from serve files. I’ll upgrade the RAM in the next few days.
I took a copy of the DB index files elsewhere, so can run those through another PC to try out. I guess once the indexing is done, I can simply copy that folder back onto the offending server to replace the broken ones?
Then indeed this:
So yeah you can kill the process.
The test more or less load the entire db into RAM. If the db is smaller than the 16GB RAM you have on the other machine, you can try there (but again, if it fills up your RAM just abort). Otherwise we need to find other ways to diagnose this.
It’s only a diagnosis, not a repair tool.
