[SOLVED] Need guidance for troubleshooting nodes that won't connect

EDIT : likely solved - db corruption.

I have 12 nodes sending to 1 host. Last night, all of them went ‘disconnected’ (at the same moment), all nodes are currently not reconnecting.

I’ve paused all but 1 of the nodes and am watching the logs on both sides of the sync, but I don’t know what I should be looking for exactly. Before I start posting logs willy-nilly, could some one indicate the most efficient way to troubleshoot/diagnose this ?

What you describe sounds like the underlying network has issues, but let’s not jump to conclusions.

As a starting point, the Web GUI has two items on the right: Listeners and Discovery with a number next to them. You may see something like 2/4. If a listener/discovery error occured, you can click on the numbers to see the error message.

This way you can check if syncthing is capable of connecting to the discovery server and/or can listen locally. Any obvious connection issues should be listed there.

syncthing_network_status

If all numbers are full/green on all your hosts then we need to look further, but this should be the first step (together with checking that all hosts have network access).

Also, some more details would be helpful (e.g which version(s) are you currently running, any upgrades performed lately, network changes, screenshots etc)

1 Like

image that looks ok, i think.

All hosts are on 1.3.3 , most are Windows, the central repo is Debian.

I was watching syslog, and this ugly thing came up :

Jan 28 12:01:42 backup-nas syncthing[14015]: panic: leveldb/table: corruption on data-block (pos=74435849): checksum mismatch, want=0x28da7a7b got=0x10745ed9 [file=2415165.ldb]

That would cause ST to continually restart…never allowing anything to connect, no ?

Sounds like syncthings database got corrupted.

I would quick-fix this by first stopping syncthing on the corrupt machine, then copying the database elsewhere as a backup (database is the entire index-v0.14.0.db folder located by default @ $HOME/.config/syncthing) and then delete/rename the folder. Next, start syncthing again - the database will rebuild itself, altough in a clean state as if you had just added all folders first time. This may cause a large/slow inital sync and possibly conflicts.

Also, check your disks just in case something is dying.

Your database is corrupt, either due to faulty hardware or due to abrupt shutdown. You’ll have to rebuild it.

yup, I was just reading over this :

reindexing 2million+ files, banzaiiii

is

stopping the syncthing service, removing dir index-v0.14.0.db , restarting service

the preferred way ?

1 Like

That’s a perfectly fine way.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.