Option to follow directory junctions / symbolic links?

I’m strongly missing an option that would allow Syncthing to follow directory junctions or symlinks.

I’m in the process of setting up Syncthing on my new notebook, and I need to synchronize many various folders scattered all over the filesystem. Under the current state of things I have no other option than to synchronize each such folder individually - which is a gargantuan task. Originally I planned to create a single folder (or several folders) that would contain directory junctions to the various scattered folders, but this doesn’t work.

I found these two threads discussing the same thing:

2 Likes

Yes, this feature does not exist, and it was deemed too hard to implement sensibly. From the conversation it feels like you are on a unix like filesystem, in which case you can probably bind-mount to achieve the same behaviour.

1 Like

I’m on Windows 10. Mounting doesn’t work there, and I do not see any other option how to do that with Syncthing.

@Catfriend1: does your hack require custom build?

@AudriusButkevicius: It seems to me simple to implement. I’ve read that in one of the linked threads you wrote

There was a discussion about this feature but it has dangerous implications when being toggled on and off on content that is already synced etc.

but I do not understand that. It seems to me that if you toggle that off, you can simply consider the folder empty or removed.

Yeah, no. That option isn’t forthcoming. Other options may or may not exist, but it’s not coming to Syncthing. Sorry.

@xarx yes, if you like, I can upload a patched exe for you. I’m using the symlink hack without problems in sendreceive and sendonly scenarios since 2016/11. But I cannot guarantee everything in there is 100% safe. Currently having deployed a patched v1.3.2 on our network.

From my point of view it’s essential to setup symlinks right and keep them forever or remove them intentionally knowing the consequenses.

This. Which is complicated by the fact that the other side can also create, change and delete symlinks via the sync connection.

1 Like

@calmh: the other side can also create, change and delete symlinks via the sync connection.

Are you talking about syncing symlinks as symlinks? Why not to consider them equal to directories, i.e. sync a symlink onto a directory?

2 Likes

Following them like they were a directory is what I did exactly. But yeah, I totally understand it from a maintainer’s point that we don’t like to put users at risk… because if anyone fails to understand the symlink follow strategy, he might be pointing fingers and blaming syncthing for his mistake. I like that idea from a technical view only, let’s say, for the tech savyy users. Duplicati for example has a config option which says by default “backup symlinks as they are” and it can be switched (by a help text and warning beside it) to the directory follow strategy.

Because we already sync symlinks as symlinks, and symlinks can also point to non-directories. (And upwards, and in chains.)

1 Like

Not only Duplicati. There are “Advanced” settings in Syncthing anyway, when anyone changes them, he must know what he’s doing.

There’s 50+ threads about this on the forum, I don’t care to repeat the discussion from all of them: https://forum.syncthing.net/search?q=follow%20symlinks

Suffice to say that following symlinks on the source of a one-shot single-direction sync program (rsync, etc.) is vastly different from following them on the source and destination in a multi-device bi-directional setup that already sync symlinks as-is. You might say “just make it an advanced option and let the user sort out the ensuing disaster” but that’s not how it usually turns out (hence this forum) and not really how we operate anyway.

There’s also the fact that I actively don’t want it, so convincing me to implement it is a fool’s errand. There’s a thousand other things I want to implement that I also don’t have time for.

1 Like

It sounds like if you burned your fingers in the past, and now you do not want to repeat that. If you “actively” refuse that, there’s hardly anything we can do about that.

I’ll have to think over what I can do with that. Syncthing has a unique synchronization mechanism that I like very much, but lacks many essential features. Hard to decide. But nothing is ideal, life is full of compromisses :-(.

1 Like

One last remark.

That’s not true, or at least not on Windows. I synced a folder containing directory junctions from Windows to NAS, and the directory junctions weren’t copied to NAS, they we simply ignored (which is good, under the current state of things). So, maybe your statement is true for Linux, but not for Windows.

Yeah you’re correct, the docs state this:

Symbolic Links (synced, except on Windows, but never followed.)

I’d say that Syncthing is tuned for Linux, and other platforms are treated with lower priority. All features specific for Linux work fine, but on Windows or Android there are restrictions.

Hmmm. Yes, but I would not call it Syncthing’s fault that Windows and Android didn’t have a clear line of feature introduction and maintenance regarding symlinks. I mean, on linux symlinks were “the same” technical thing from the beginning. Windows did introduce them “hidden” with xp/ntfs and changed their behaviour and interpretation over time, recently switching over to some reparse points as an addition to their borked symlink handling.

2 Likes

I’m not saying whose fault it is, it’s just an observed fact. For instance, Syncthing cannot write SDcards on Android, and - as far as I learned - it’s fault of the GO language, as Android is not a primary platform for them. But if Android was a primary platform for Syncthing, there are possible workarounds, though they require a lot of work.

Concerning Windows, I’d say that the main usage of directory junctions is the same since Windows Vista - look how they are used inside user profile and system directories. Of course, there was a technical evolution, also there are several types of symlinks in NTFS, but the basic usage is IMO still the same.

So, I’ve took some time to play around with my little go knowledge and the Syncthing code. Got it to follow symlinks during the scan.

Source: https://github.com/Catfriend1/syncthing/pull/5/files

It’s correctly stating the folders and files beyond symlinks on Windows and Linux. To make it functional, TraverseSymlink checks have to be disabled, too, like in https://github.com/Catfriend1/syncthing/commit/441ef5cedf5ec727475b628f1f2a617089817d88 .

I don’t feel stable with the hack, but will do manual testing if it works because I’m curious.

1 Like

I’m not sure where you’re going with this, so not sure how to respond. Is this an experiment for your personal edification? Those are great. Are you recommending someone use this or aiming for inclusion in Syncthing? If so, my first reaction is that this probably allows another device to read and write files anywhere on the system, so that’s not something I’d recommend.

3 Likes

Side track - you can take a snapshot and mount that snapshot where you like, then destroy it. Many advantages over copying.

1 Like