[Note: This was originally submitted as a bug report, but Audrius was quick to close it and suggested that the support forum would be a better place for it. Thus this post. However, please note that I have already resolved the mess that Syncthing created, so I’m not actually seeking any help with my setup. I’m just posting this as a record of how things can go wrong – very wrong indeed – after a disk replacement containing Syncthing’s database and part of its synced dataset. The replacement and the restoration process from backup was without any reported errors and I have not noticed any problems with any other services or files on my server so I see not reason to believe that Syncthing’s database somehow got corrupted during the restoration. Rather, it seems that the change of disks itself confused it mightily or perhaps the database was in an incomplete state when it got backed up. If anyone has some better theories, I’m all ears. Thanks.]
This is a report of how Syncthing managed to mix up all the synced files on one of my servers and spread them to every other folder to every other instance that it was syncing with. Yes, really!
The main SSD of Mega, my Mac mini server, failed on me a few weeks ago. Fortunately, it was under warranty, so I recently received a replacement SSD and restored the contents from backup. After the backup was complete and the server was rebooted, Syncthing started up on Mega and quickly ran out of open files. Strange, I thought, it had never done that before. Still, I just up’ed the max number of open files from 256 to 1024 and let it continue.
The next day, I picked up Neo, my MacBook Pro laptop, and noticed that it was out of disk space. Huh, what had happened? A closer inspection showed that every folder that was being synced with Mega was full of files – in fact every synced folders seemed to have a copy of all the files from all the folders that Mega was syncing with any other host!
That is, if Mega had folders A, B, and C, and Neo was syncing A and B with Mega, Neo’s A and B folders now both appeared to contain all the files from A, B, and C all mixed together. The same was true for the A folders on all the other hosts that was syncing with Mega and/or Neo.
Not only has this created a mess of epic proportions, it’s also a giant security hole since the private files in B now have been distributed to a whole slew of public hosts that only were supposed to have the A files.
And as if that wasn’t bad enough, when I frantically started deleting the misplaced files and folders, Syncthing started putting them back again!
That is, when I started deleting the B files from the A folder, Syncthing promptly recreated them again! At first I thought I had made a mistake when deleting them, but when I deleted them again and listed the files in the directory, they were clearly gone – only to reappear a few seconds later.
To be fair, it seems to mostly(? only?) be the subdirectories in which the files reside, not necessarily the files themselves – I wasn’t paying full attention when it first happened, and now when I’m trying to reproduce it, I’m only seeing (empty) subdirectories being recreated. Looking at syncthing.log, I see a lot of:
[SNPSJ] 12:24:12 INFO: Puller (folder A, item “bsubdir/bfile”): no connected device has the required version of this file
This seem to cause the recreation of bsubdir.
The only solution I’ve been able to come up with is to delete all Syncthing configurations on all hosts and recreate them from scratch as just rescanning the folders seems to make no change. I’ve put aside a copy of the config files on Mega together with syncthing.log. I had a quick look at syncthing.log from after the restoration, but I couldn’t find anything obvious in it except for a whole bunch of “too many open files” errors. I can make them available if it would help, but I would first need to anonymize the contents.
Syncthing Version v1.1.4 running under macOS 10.14.5 + Ubuntu 16.04.5 LTS & 18.04.2 LTS.