# panic: non-increasing sequence
**URL:** https://forum.syncthing.net/t/panic-non-increasing-sequence/14570
**Category:** Support
**Created:** [February 26, 2020, 8:54am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570 "2020-02-26T08:54:03Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 26, 2020, 8:54am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/1 "2020-02-26T08:54:03Z")
Hi there
My Syncthing has been down for a couple of weeks and I didn’t realise. I’m seeing -
Panic at 2020-02-26T00:25:42Z panic: non-increasing sequence, current: 976 \<= previous: 8696
This is 1.4.0 rc-7
Can anyone advise what I should do?
Thank you 🙂
---
### Author:  [@Moisie](https://forum.syncthing.net/u/Moisie)
#### Post date: [February 26, 2020, 9:29am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/2 "2020-02-26T09:29:50Z")
Hello:
Likewise, I’m seeing this on one of my machines the last couple of days.
The only change I recently made was to upgrade the SynoCommunity ST package - which would have temporarily downgraded the Syncthing binary. Since then, I’ve seen it panic with this error twice.
Running Syncthing 1.3.4 - I can provide a log privately if required.
Thanks.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [February 26, 2020, 9:44am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/3 "2020-02-26T09:44:53Z")
So, it’s a bad sign that this happens and a log might be useful, but the check is also only performed if you have model tracing on. Turn that off for normal operations.
---
### Author:  [@Moisie](https://forum.syncthing.net/u/Moisie)
#### Post date: [February 26, 2020, 9:51am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/4 "2020-02-26T09:51:23Z")
Thanks for the quick reply Jakob.
You’re right - I had model tracing on to debug an issue we’ve been having with deleted files re-appearing - but I’ll turn it off now.
I’ll send you the logs privately, just in case they’re of any use for you.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [February 26, 2020, 9:58am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/5 "2020-02-26T09:58:17Z")
In 1.3.4 I’m not really surprised by this.
In 1.4.0-rc.7 it’s more surprising because that version attempts to fix this as part of the upgrade migration. Of course database problems can happen anyway due to flaky hardware or whatnot, but …
---
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 26, 2020, 11:19am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/6 "2020-02-26T11:19:28Z")
Hi there
I don’t think I have model tracing on. I’m running Syncthing via SyncTrayzor on Win10. I’ve looked in the ST interface but see nothing re tracing.
WOuld it help for me to send you a panic log direct?
---
### Author:  [@imsodin](https://forum.syncthing.net/u/imsodin)
#### Post date: [February 26, 2020, 11:33am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/7 "2020-02-26T11:33:48Z")
@calmh Discussion already ongoing in the PR ~~We only check for missing and duplicate sequence entries, but not that sequence keys match sequences in file infos. I made a function that checks for that and fixes it, activatable by env var. Could you have a look to see if it is sane (I did some testing…):~~ [https://github.com/syncthing/syncthing/pull/6367](https://github.com/syncthing/syncthing/pull/6367)
@marky_uk Go to the menu at actions\>logs\>debug facilities and check if model is ticked. Also would you be ok to run a custom syncthing build to see if the check I wrote about above fixes the problem for you?
---
### Author:  [@imsodin](https://forum.syncthing.net/u/imsodin)
#### Post date: [February 26, 2020, 11:43am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/8 "2020-02-26T11:43:22Z")
@calmh just told me that our existing index checker can detect this problem. Could you please download the `stindex` tool for your system from [https://build.syncthing.net/buildConfiguration/Syncthing\_Tools/55765?buildTab=artifacts](https://build.syncthing.net/buildConfiguration/Syncthing_Tools/55765?buildTab=artifacts) (you can login as guest), stop Syncthing and then run `stindex -mode idxck` and report the output here - that would be very helpful!
---
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 26, 2020, 11:44am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/9 "2020-02-26T11:44:31Z")
Boo -
---
### Author:  [@imsodin](https://forum.syncthing.net/u/imsodin)
#### Post date: [February 26, 2020, 12:24pm UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/10 "2020-02-26T12:24:55Z")
Upsi. Low memory system with lots of data in Syncthing?
If yes, any chance you can copy the index-v0.14.0.db directory to a machine with more ram and run the test there? Or upload it somewhere so we can inspect?
---
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 26, 2020, 1:25pm UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/11 "2020-02-26T13:25:16Z")
Hiya
So sorry to ask. Managed to copy the files to another PC that doesn’t have syncthing installed, and downloaded the stindex file to the same folder. When running the command I’m shown - CreateFile c:\Users(path here)\index-v0.14.0.db: The system cannot find the path specified. (is another instance of Syncthing running?)
That path doesn’t exist as I don’t have Syncthing on this PC. Do I need it installed? Or create the path? Or supply params? Or something else? Thank youuu!
---
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 26, 2020, 1:39pm UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/13 "2020-02-26T13:39:09Z")
Hey - on the new PC, I get the same failure:
It’s got 16GB RAM, and is using about half that currently.
---
### Author:  [@imsodin](https://forum.syncthing.net/u/imsodin)
#### Post date: [February 26, 2020, 1:42pm UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/14 "2020-02-26T13:42:18Z")
Forgot about that: If it’s not at the default location, you need to specify the path. So e.g. for `\foo\bar\index-v0.14.0.db` call `stindex -mode idxck \foo\bar\index-v0.14.0.db`.
> [@marky\_uk](#):
>
> It’s got 16GB RAM, and is using about half that currently.
You downloaded the 32bit binary (ams\_386 in the screenshot), while with 16GB ram you are definitely on a 66bit system. You need to download the windows-amd64 variant.
I assumed that was on purpose on the original system, maybe it would have worked there too if it’s 64bit as well.
---
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 26, 2020, 1:42pm UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/15 "2020-02-26T13:42:53Z")
What a muppet I am - I’ll try with the larger file and the params!
---
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 27, 2020, 8:36am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/16 "2020-02-27T08:36:44Z")
Hiya. Running this on the original PC with 8GB RAM. How long do you think this would normally take? It’s been running since yesterday and still going!
---
### Author:  [@imsodin](https://forum.syncthing.net/u/imsodin)
#### Post date: [February 27, 2020, 8:44am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/17 "2020-02-27T08:44:51Z")
I was wondering what’s going on, with you responding laser-fast usually 🙂
How much data do you have in Syncthing? And what does the RAM usage look like?
I just ran it locally on my production env with ~500k files and ~0.5TB data, with database on an SSD in ~10s.
---
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 27, 2020, 9:21am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/18 "2020-02-27T09:21:18Z")
Haha, I debated replying and wondered about forum etiquette - do people want to know “it’s still running” and receive boring updates, or wait until it’s done but not know if I’ve gone to Wuhan for the weekend.
Data = 1.7TB, 4.8M files. RAM usage = 5.2GB by the indexing process.
---
### Author:  [@calmh](https://forum.syncthing.net/u/calmh)
#### Post date: [February 27, 2020, 9:22am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/19 "2020-02-27T09:22:08Z")
If your system starts swapping due to the index checker it’ll probably take forever.
---
### Author:  [@marky\_uk](https://forum.syncthing.net/u/marky_uk)
#### Post date: [February 27, 2020, 9:26am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/20 "2020-02-27T09:26:54Z")
Yeah, RAM is at 92% usage. I chose such a low spec Dell for the “server” as normally it doesn’t do much apart from serve files. I’ll upgrade the RAM in the next few days.
I took a copy of the DB index files elsewhere, so can run those through another PC to try out. I guess once the indexing is done, I can simply copy that folder back onto the offending server to replace the broken ones?
---
### Author:  [@imsodin](https://forum.syncthing.net/u/imsodin)
#### Post date: [February 27, 2020, 9:33am UTC](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570/21 "2020-02-27T09:33:07Z")
> [@marky\_uk](#):
>
> Yeah, RAM is at 92% usage.
Then indeed this:
> [@calmh](#):
>
> If your system starts swapping due to the index checker it’ll probably take forever.
So yeah you can kill the process.
> [@marky\_uk](#):
>
> I took a copy of the DB index files elsewhere, so can run those through another PC to try out.
The test more or less load the entire db into RAM. If the db is smaller than the 16GB RAM you have on the other machine, you can try there (but again, if it fills up your RAM just abort). Otherwise we need to find other ways to diagnose this.
> [@marky\_uk](#):
>
> I guess once the indexing is done, I can simply copy that folder back onto the offending server to replace the broken ones?
It’s only a diagnosis, not a repair tool.
[Next page](https://forum.syncthing.net/t/panic-non-increasing-sequence/14570.md?page=2)