What if I create 2 identical directory structures with files and delete a single file in one, THEN bring sync on line on both ends. How does SyncThing det’n to propagate the file to both, or delete it on both?
I don’t think the OS keeps track of when deletions happen, so how does SyncThing decide which is more recent - the deletion or the file add ? In the man pages, descriptions under “Syncing” don’t address the delete issue, unless (am I getting this right?) that when the “global version” goes to nothing, then a deletion is what each device strives to update to? That still doesn’t answer the question of what happens when two directories are ab-inito different, or how a “deletion” becomes the target.
The threat is that when the device is unmounted, the underlying empty directory is sync’d to all peers. Ouch! Lots of data deletion! I’ve read the debate in the prior post. IMHO, the MOST important thing is SyncThing MAY NOT silently delete all my files on all my peer systems. To say, “User should know what he’s doing and not unplug the USB” is really scary to me.
Syncthing tries to protect help this issue with the .stfolder file (if it’s missing assume a dismount and do not propagate this as deletions to the rest of the peers). Okay, but that invalidates using a USB mount or fuse mount on any subdirectory in the sync.
Are you aware of how Unison does this? Maybe this should be implemented. There is a “usbmount” directive that can be associated with any path. How about implementing a SyncThing option “-transientmount=”, where can be “./” or “./anysubdirectory”? Then, the sync program checks the mount table to be sure there IS a mount at that location before it does anything with that directory. Sounds easy conceptually, but I’m not sure what it would take to program. If this issue is the only reason you use the .stfolder, then the entire .stfolder logic thread can be deleted from SyncThing.
My real goal here is to use encfs (with the --reverse parameter) and sync peer-2-peer encrypted directories that are really virtual mounts switched on and off with encfs. The real data is stored clear text on the computer and the sync directory is really an empty mount point for encfs to show the encrypted version of the files.
In general the behaviour on conflicts is explained in the FAQs. In short, the modification always wins over deletion.
Having a drive mounted in a subdirectory of a Syncthing folder is indeed a bad idea for exactly the reason you explained. So currently it is only save to create a syncthing folder on the mount path or in a child directory of it. For your use case, this should fit your use case, shouldn’t it?
I think the problem with the “Unison-way” is, to check for a mount in a cross-compatible way.
FAQ says, “If the conflict is between a modification and a deletion of the file, the modified file always wins and is resurrected without renaming on the device where it was deleted.”
But that’s not true for the sync-sub-directories, which is why I asked the question. Instead, when the overlaid mount is removed and the naked directory under it is exposed, all the files are deleted on all the peers. ! This directly violates “Design Principal #1” of Syncthing (keeping user data safe).
If we accept the limit “You may not have overlaid mounts anywhere under a sync directory,” that’s a pretty bad limitation. And if it ever accidentally gets violated, we loose all the user data? Ouch!! I can’t trust a system with such a potent latent problem.
The unison developers thought through how to handle USB mounts and dismounts and I think that’s an almost perfect match to overlaid mounts. In the linux world, I think mounts would show in the mount table. In the windows world, the concept is not a threat because there are no overlaid mount points - just drive letters that pop into existence or not.
…the reason I’m trying to do sync with subdirectory mounts is to build a tool chain to answer the Syncthing #1 feature request: encrypted syncs. This is already possible with encfs “forward” syncs, but with “reverse” syncs, the encrypted data is virtual so it pops in and out of existence based on when ecnfs is running. I can’t have a dismount delete all of my files on all the servers if I unintentionally stop encfs with the sync running.
The reason I don’t want to mount and dismount at the sync root level is because the root level of my sync contains UNencrypted tools like encfs, Synthing, TrueCrypte, BTSync, 7zip, etc. Of course, they have to be cleartext so I can access the encrypted data. I also want my TrueCrypte volumes to live at that level, because there is no reason to double-encrypt them. I need the data root directory of my encfs encryption tool to be one layer down.
We handle his by looking for the .stfolder file in the folder root. If this is gone, we will interpret it as a mount failure or similar and stop the sync. If it is still present but other files are gone, we interpret it as an intentional delete. It’s not obvious to me how we would do this better in an automatic, cross platform way - but I’m open to suggestions.
Your setup with encfs is inherently risky from this standpoint, as you’ve noted.
That’s because it is talking about file conflicts. Deleting or unmounting an entire directory is not necessarily a conflict, there would need to be a simultaneous change on these files on another node.
This in not trivial to solve in a cross platform way. If you can put together a PR that is elegant I am sure the core ST team will merge it.
Otherwise you run the risk of the data being deleted when the folder is empty. You haven’t lost the data… it is still on the machine that the mount was removed from.
Syncthing has acted as expected and made the directory empty on the other nodes as well.
Mount a USB drive inside a Windows folder instead of a new drive letter? I can’t remember ever seeing this done – I learn something new every day. Is there an easy way you can describe to me using OS tools rather than a special program I need?
.stfolders in every subdirectory is gross; must be something better. Why not allow a command line option to simply name a path that is to be treated special? Below is copied from the Unison manual. It seems easy to implement (at least logic wise). I don’t understand Syncthing internal structures, but it seems the status “nothing in this directory” should already be known by Syncthing, right?
"Mount Points and Removable Media
Using Unison removable media such as USB drives can be dangerous unless you are careful. If you synchronize a directory that is stored on removable media when the media is not present, it will look to Unison as though the whole directory has been deleted, and it will proceed to delete the directory from the other replica—probably not what you want!
To prevent accidents, Unison provides a preference called mountpoint. Including a line like “mountpoint = foo” in your preference file will cause Unison to check, after it finishes detecting updates, that something actually exists at the path foo on both replicas; if it does not, the Unison run will [not sync that directory]."
Inside the Windows disk management (diskmgmt.msc) you can right click on a drive and choose Change Drive Letters and Paths….
There you can assign as many letters as you want (even 0), as long as you have free ones ;). And you can set one or multiple folders where to mount the drive.
In the past I always used this to remove the annoying drive letters of build in multi card readers.
I can imagine this being done the way you describe: Specify such a special mount path somehow in configuration, then syncthing tracks its status in a similar way as it does the root folder with .stfolder (maybe call it .stsubfolder) and ignores that subfolder if this file is missing. Even if there is no inherent drawback in such a method, this is a corner case and I imagine the only way to get it into Syncthing is by doing the PR yourself.
I could see this possibly being done by allowing there to be more than one “.stfolder” and with a configurable name. I think there’s an issue for the name already, somewhere in the tracker. You could then say that the folder markers are {.stfolder, foo/bar/shouldAlwaysExist} and if either of those are missing the folder is stopped.
The thread was sidetracked. I want to jump back to my first question. The FAQ says, “If the conflict is between a modification and a deletion of the file, the modified file always wins and is resurrected without renaming on the device where it was deleted.”
What about when Syncthing is first-time started with an extra file on one node. Nothing has been “modified”. Is the file propagated or deleted?
Also, if the FAQ is correct, how can Syncthing propagate a legitimate delete operation?
There is a difference between a file not being present and a file having been deleted. Syncthing considers something “deleted” when it has itself observed the file first existing and then no longer existing.
Turns out a volume mounted by encfsMP under Windows is not accessible via the diskmgmt program, so dskmgmt can’t be used to change the drive letter or mount it at another directory.
So… I think in the context of using Syncthing with EncfsMP, there are “no overlaid mount points - just drive letters that pop into existence or not.”