I have 12 nodes sending to 1 host.
Last night, all of them went ‘disconnected’ (at the same moment), all nodes are currently not reconnecting.
I’ve paused all but 1 of the nodes and am watching the logs on both sides of the sync, but I don’t know what I should be looking for exactly.
Before I start posting logs willy-nilly, could some one indicate the most efficient way to troubleshoot/diagnose this ?
What you describe sounds like the underlying network has issues, but let’s not jump to conclusions.
As a starting point, the Web GUI has two items on the right: Listeners and Discovery with a number next to them. You may see something like 2/4. If a listener/discovery error occured, you can click on the numbers to see the error message.
This way you can check if syncthing is capable of connecting to the discovery server and/or can listen locally. Any obvious connection issues should be listed there.
If all numbers are full/green on all your hosts then we need to look further, but this should be the first step (together with checking that all hosts have network access).
Also, some more details would be helpful (e.g which version(s) are you currently running, any upgrades performed lately, network changes, screenshots etc)
I would quick-fix this by first stopping syncthing on the corrupt machine, then copying the database elsewhere as a backup (database is the entire index-v0.14.0.db folder located by default @ $HOME/.config/syncthing) and then delete/rename the folder. Next, start syncthing again - the database will rebuild itself, altough in a clean state as if you had just added all folders first time. This may cause a large/slow inital sync and possibly conflicts.
Also, check your disks just in case something is dying.