I’m strongly missing an option that would allow Syncthing to follow directory junctions or symlinks.
I’m in the process of setting up Syncthing on my new notebook, and I need to synchronize many various folders scattered all over the filesystem. Under the current state of things I have no other option than to synchronize each such folder individually - which is a gargantuan task. Originally I planned to create a single folder (or several folders) that would contain directory junctions to the various scattered folders, but this doesn’t work.
I found these two threads discussing the same thing:
Yes, this feature does not exist, and it was deemed too hard to implement sensibly.
From the conversation it feels like you are on a unix like filesystem, in which case you can probably bind-mount to achieve the same behaviour.
Suffice to say that following symlinks on the source of a one-shot single-direction sync program (rsync, etc.) is vastly different from following them on the source and destination in a multi-device bi-directional setup that already sync symlinks as-is. You might say “just make it an advanced option and let the user sort out the ensuing disaster” but that’s not how it usually turns out (hence this forum) and not really how we operate anyway.
There’s also the fact that I actively don’t want it, so convincing me to implement it is a fool’s errand. There’s a thousand other things I want to implement that I also don’t have time for.
It sounds like if you burned your fingers in the past, and now you do not want to repeat that. If you “actively” refuse that, there’s hardly anything we can do about that.
I’ll have to think over what I can do with that. Syncthing has a unique synchronization mechanism that I like very much, but lacks many essential features. Hard to decide. But nothing is ideal, life is full of compromisses :-(.
That’s not true, or at least not on Windows. I synced a folder containing directory junctions from Windows to NAS, and the directory junctions weren’t copied to NAS, they we simply ignored (which is good, under the current state of things). So, maybe your statement is true for Linux, but not for Windows.
I’d say that Syncthing is tuned for Linux, and other platforms are treated with lower priority. All features specific for Linux work fine, but on Windows or Android there are restrictions.
I’m not saying whose fault it is, it’s just an observed fact. For instance, Syncthing cannot write SDcards on Android, and - as far as I learned - it’s fault of the GO language, as Android is not a primary platform for them. But if Android was a primary platform for Syncthing, there are possible workarounds, though they require a lot of work.
Concerning Windows, I’d say that the main usage of directory junctions is the same since Windows Vista - look how they are used inside user profile and system directories. Of course, there was a technical evolution, also there are several types of symlinks in NTFS, but the basic usage is IMO still the same.
I’m not sure where you’re going with this, so not sure how to respond. Is this an experiment for your personal edification? Those are great. Are you recommending someone use this or aiming for inclusion in Syncthing? If so, my first reaction is that this probably allows another device to read and write files anywhere on the system, so that’s not something I’d recommend.
Can you please elaborate this statement, or at least link to another discussion here where this was explained? Because I don’t see that’s true. Otherwise the whole NTFS links mechanism would be a security breach in Windows, because your argument would most probably apply to many situations where they are used, not just Syncthing. Thanks.
We don’t create symlinks on Windows so from a Syncthing perspective this maybe isn’t relevant there.
The patch disables our “don’t follow symlinks” protections. In a default setup I can then create a symlink foo -> /home, wait for you to sync it. Next scan you’ll follow the symlink and announce all of /home for me to sync under the foo “directory”. (Before it crashes due to following the symlink again, recursively, probably.)
Following symlinks safely is not as simple as just disabling our protections. Please don’t just throw out patches and expect me to explain why they don’t work.
You’ve disabled the safety grip on the chainsaw so that it can run without anyone holding it. That doesn’t make it a robotic lawnmower, it just makes it a really unsafe chainsaw.
OK. From my point of view, following symlinks is fundamentally incompatible with letting other devices influence symlinks. Given that, I guess a minimal read-only implementation would be
Following symlinks can only ever happen on send-only folders
We must not allow nesting folders (which we otherwise do allow)
Changing the folder type must disable following symlinks
So what happens with the data already synced out there? It must be marked as deleted, which might be unexpected.
Handle circular and chained symlinks (foo -> ., foo -> .., foo -> bar -> foo, foo -> foo etc)
“Handle” by stopping the folder on detection? It’s not obvious what the user intended to happen here I think.
Symlinks can be to files or directories or specials (/dev/random)
Consider whether allowing symlinks within a folder are a good idea or not.
Symlinks can be dangling, do we represent that somehow or is it a scan failure (stop folder)?
If not a scan failure, what happens to remote data when a symlink suddenly becomes dangling?
We use .stfolder to detect against unmounted storage being interpreted as a full delete. Can we do something similar when following symlinks to other partitions?
Probably more corner cases, that’s just from the top of my head. Handling symlinks on a send-receive folder is much more complicated. You’d have to at minimum disable processing incoming changes that affect symlinks, handle the case where a symlink points to a file on a different partition and where we would put the temp file when syncing a change to that file, etc.
A safer variation might be one we’ve explored before, where symlinks to follow must be explicitly named in the folder config. That doesn’t invalidate any of the above checks, but makes it safer to use in practice because inadvertently adding a new symlinks somewhere to cause a circular reference won’t be an error that needs to stop the folder, etc. I think that experiment failed because I tried to make it read/write but maybe it might be viable as read-only.
I’d say that this scenario cannot be performed on Windows, because directory juctions seem to be completely ignored by Syncthing. When I create a directory juction, it is not propagated to a linux machine. I haven’t tried to sync Windows with Windows, hence I don’t know whether directory junctions propagate there (but you wrote that ST dosn’t write symlinks on Windows). If they don’t, then there is IMO no reason to distinguish directory junctions from normal directories. But, of course, there are also other types of symlinks in Windows, and I haven’t tested yet, how Syncthing handles them.
If directory junctions are completely ignored, as written above, there’s IMO no need for this restriction. I don’t know whether other types of symlinks are handled by ST differently and whether Golang is able to distinguish different types of Windows symlinks.
All in all, if Windows symlinks are never written (as symlinks; currently it seems they are ignored), then following them and writing them as directories should be safe, IMO.
Edit: Of course, you can create a recursive structure using symlinks on your Windows computer, or point them to sensitive directories. But this is your own responsibility and depends on your privileges.