Massive delete at first sync...


#1

I guess some of this could go in the support forum. I did not write every step I took so it won’t be easy to reproduce, hence the “User Story”

Here is the deal. I had one instance with “Local state Files 6729 Dirs 1476” under /var/autofs/Files/foo, share named “Bar” (there is a foo/bar subdirectory) It was waiting to share with a mirrored machine.

The mirror machine is a clone. I cloned the OS and the data. I took special care of deleting all .pem files in the case of Syncthing so that the machine has its own identity. That instance had the same configuration, “Local state Files 6729 Dirs 1476” under /var/autofs/Files/foo, share named "Bar" and was waiting to connect to the 1st machine.

Then the “Folder ID” thing was invented, and the 2 machines had a different folder ID for the same path and same data.

Then I connected the machines. I had the great idea to put the first machine, which is in production, in send-only mode.

Then I sent the “Bar” share from the 1st machine to the second one. There was a wimpy warning notice that the receiving path was a subdirectory of an existing share on the 2nd machine. It was not a subdirectory. It was a full clone share already defined on that machine. The folder ID was different.

So I hit save and all my local files were gone. So much for avoiding the initial transfer. I don’t know what would have happened if I did not set the 1st instance to read-only.

The “Folder ID” comment on a share reads "Required identifier for the folder. Must be the same on all cluster devices." FTW are “cluster devices”, now? Shouldn’t the comment rather read “SAME FILES BUT UNDER ANOTHER FOLDER ID WILL BE DELETED!!”?

It seems obvious to me that in case people want to share a large amount of data, they will if possible bootstrap the mirror by making a local copy before setting up the mirror. It would be good if a) Syncthing would somehow refrain from removing 100% of the files in an existing share without warning, b) a how-to was added to the documentation.


(Jakob Borg) #2

You also cloned Syncthings index database. When the files that according to the index should be there are then not there, the conclusion is that you deleted them.


#3

The paths were exactly the same, though?


(Jakob Borg) #4

Maybe I didn’t follow along correctly; there are a few curves along the way with changing paths and folder ids and whatnot. Suffice to say, this is not safe unless you take extreme care.


#5

Right. So I stand by my recommendation: have a “how-to clone” for people who want to bootstrap a large mirror. Large syncs can be slow and large deletes are disastrous.


(Audrius Butkevicius) #6

There is no how-to as we don’t recommend doing what you did.


(Jakob Borg) #7

If I were to make one recommendation, it’s to set one up-to-date device to “send only” while doing shady stuff. Hit “override” as appropriate. Once everyone is happy, set it to normal mode.


#8

Yeah. But when you have a few gigs of data and a slow link, starting with a local copy on both sides is what anybody would do, I think.


#9

Hear, hear. Send-only saved my bacon, indeed. (now working at making everybody happy, after more “shady” stuff.)


(Simon) #10

Local copy is perfectly fine, people do that all the time, it won’t delete your data. You seemed to have meddled with copying indexes, that is the dangerous part.


#11

This I don’t understand. The files were exactly in the same paths on both machines. Does the index discriminate much further than that?


(Audrius Butkevicius) #12

In that case nothing should be removed. Can reliably reproduce it?


#13

Not sure I can. But I seem to have a knack for falling in that trap. If I have time I will setup a reproducible test.

(And the broken mirror is repaired and all in sync by now :slight_smile:


(Tom Atkinson) #14

Moral of the story: always remove the share in Syncthing before say… deleting all the files from it. This will prevent the other machine having all it’s files deleted. Tricky but makes sense in a weird way.