Syncthing not running after latest update 1.2.6

BTW, for perspective, my Syncthing deployment is a PastPerfect dataset that is 10 TiB across more than a million files, the indexes are all larger than 10 GiB, and most of the TrueNAS nodes have 8 GiB of physical memory.

Per comments at Github, Syncthing 2.0 could be a bumpy upgrade, but let me nevertheless express gratitude that the Syncthing team makes good software. Years-ago I ran trials with competing products, and only Syncthing could handle big data across relatively slow networks and small machines.

3 Likes

Hi Darik,

Thanks for this information.

As a newbie to this, could you please tell me where (and how) I would put these two lines which you shared?

Thanks so much for your patience,

Kevin

In the TrueNAS web interface, you run those removal commands at System → Shell.

You might also need to run sudo -i first depending on your system configuration.

Doing this kind of thing without understanding it often causes catastrophic data loss. Be well and be lucky if you choose this solution.

Thanks so much.

I’m used to doing this kind of thing on my Ubuntu machine. Just wasn’t sure where to put it in TrueNAS.

Thanks for your help,

Kevin

Am I correct in assuming that Syncthing has gotten on top of this bug, because an update has just been released?

It’s unclear what, if anything, is a Syncthing bug here. There will be no relevant change in 2.0.1.

1 Like

Thanks for this. I see you just basically deleted the existing database and started the rebuild.

As I have my appdata on a hostpath, the exact shell commands need to obviously be modified for any folks that don’t use the default ix- structure.

Upon deleting the existing db and restarting the app, the rebuild started and should be done in a couple days! Greatly appreciate the support.

Mine has been on this same message for 2 days (25TB approx)

Will re-installing fix it, or will it still have to convert?

On TrueNAS 25.04+ the middleware/docker silently aborts syncthing migration operation.

See my other post from before I found this thread, or this github issue with a very hacky workaround. Deleting old database works as well, but I wanted to prevent that.

As for logs from syncthing: the migration is reported as running (ref logs here), but no new messages keep being logged, which in turn means the migration is not running at all.

If Syncthing replied on /rest/noauth/health while migration is in progress, it might mitigate this issue, but I have not tested it. The internal test command is plain: wget --quiet --spider http://127.0.0.1:8384/rest/noauth/health. @calmh maybe this^ would help.

This seems like a compound bug:

  • Nothing in the TrueNAS webui or logs indicates that the docker container is misbehaving or that the middleware manager is killing processes. I would expect a red blinky in the Alerts list
  • Even if TrueNAS kills the syncthing parent, its children persist and are actively doing I/O.
  • Nothing in the syncthing log plainly indicates that it was interrupted.

Syncthing runs as two processes, one monitor and one worker child. I don’t know truenas so I’m just speculating, but if it kills the monitor process abruptly (like sigkill instead of sigterm) while also not killing the entire process group, that would leave the worker child running as an orphan while logs go to dev null. Not a great thing to do really.

1 Like

Has there been some type of official/approved method to fix this yet? I’d really like to get this running.

I’ve had the same error and the migration was making no progress in nearly 24 hours. I decided to stop it, rename the old and the new db to .bak and then start syncthing and let it index again.

A pain but at least it will be on SQLlite then and hopefully no more db changes going forward.

Not sure what you mean by ā€œrename the old and the new db to .bak and then start syncthing and let it index again.ā€, i looked in the .STfolder , but was empty, could you possibly elaborate for a newby?

If i just delete the .stfolders will syncthing just rebuild?

Mines been running 3 days with zero progress

I used SSH to open Midnight Commander and navigate to the config folder of my syncthing instance.

in there there was say the index-v14 db and the new index-v2 database.

used F6 to add .bak after the end of both folders.

I then started syncthing again and it’s currently still indexing but using the SQLlite db now, once indexed I will just delete both of the .bak db I renamed.

1 Like

ohh, bit above my paygrade i think !! Looks like im re-syncing this week. Thanks for the help @maddogspeighty