Option to follow directory junctions / symbolic links?

Feature
1

I’m strongly missing an option that would allow Syncthing to follow directory junctions or symlinks.

I’m in the process of setting up Syncthing on my new notebook, and I need to synchronize many various folders scattered all over the filesystem. Under the current state of things I have no other option than to synchronize each such folder individually - which is a gargantuan task. Originally I planned to create a single folder (or several folders) that would contain directory junctions to the various scattered folders, but this doesn’t work.

I found these two threads discussing the same thing:

2 Likes
2

Yes, this feature does not exist, and it was deemed too hard to implement sensibly. From the conversation it feels like you are on a unix like filesystem, in which case you can probably bind-mount to achieve the same behaviour.

1 Like
3

Hi, I’ve already made a dirty hack to follow symlinks on Windows ( https://github.com/Catfriend1/syncthing/releases/tag/v1.3.2-symlink-support ). But it didn’t work for linux. Because I need this too, I’ve planned to try it with imsodin’s tips I got via PM when time allows it to play with the code (I’m not familiar with go.).

Edit: mount -bine is a good advice for linux . It has just one disadvantage that it does not work with ZFS (tried it already).

4

I’m on Windows 10. Mounting doesn’t work there, and I do not see any other option how to do that with Syncthing.

@Catfriend1: does your hack require custom build?

@AudriusButkevicius: It seems to me simple to implement. I’ve read that in one of the linked threads you wrote

There was a discussion about this feature but it has dangerous implications when being toggled on and off on content that is already synced etc.

but I do not understand that. It seems to me that if you toggle that off, you can simply consider the folder empty or removed.

5

Yeah, no. That option isn’t forthcoming. Other options may or may not exist, but it’s not coming to Syncthing. Sorry.

6

@xarx yes, if you like, I can upload a patched exe for you. I’m using the symlink hack without problems in sendreceive and sendonly scenarios since 2016/11. But I cannot guarantee everything in there is 100% safe. Currently having deployed a patched v1.3.2 on our network.

From my point of view it’s essential to setup symlinks right and keep them forever or remove them intentionally knowing the consequenses.

7

This. Which is complicated by the fact that the other side can also create, change and delete symlinks via the sync connection.

1 Like
8

@calmh: the other side can also create, change and delete symlinks via the sync connection.

Are you talking about syncing symlinks as symlinks? Why not to consider them equal to directories, i.e. sync a symlink onto a directory?

2 Likes
9

Following them like they were a directory is what I did exactly. But yeah, I totally understand it from a maintainer’s point that we don’t like to put users at risk… because if anyone fails to understand the symlink follow strategy, he might be pointing fingers and blaming syncthing for his mistake. I like that idea from a technical view only, let’s say, for the tech savyy users. Duplicati for example has a config option which says by default “backup symlinks as they are” and it can be switched (by a help text and warning beside it) to the directory follow strategy.

10

Because we already sync symlinks as symlinks, and symlinks can also point to non-directories. (And upwards, and in chains.)

1 Like
11

Not only Duplicati. There are “Advanced” settings in Syncthing anyway, when anyone changes them, he must know what he’s doing.

12

There’s 50+ threads about this on the forum, I don’t care to repeat the discussion from all of them: https://forum.syncthing.net/search?q=follow%20symlinks

Suffice to say that following symlinks on the source of a one-shot single-direction sync program (rsync, etc.) is vastly different from following them on the source and destination in a multi-device bi-directional setup that already sync symlinks as-is. You might say “just make it an advanced option and let the user sort out the ensuing disaster” but that’s not how it usually turns out (hence this forum) and not really how we operate anyway.

There’s also the fact that I actively don’t want it, so convincing me to implement it is a fool’s errand. There’s a thousand other things I want to implement that I also don’t have time for.

1 Like
13

It sounds like if you burned your fingers in the past, and now you do not want to repeat that. If you “actively” refuse that, there’s hardly anything we can do about that.

I’ll have to think over what I can do with that. Syncthing has a unique synchronization mechanism that I like very much, but lacks many essential features. Hard to decide. But nothing is ideal, life is full of compromisses :-(.

1 Like
14

One last remark.

That’s not true, or at least not on Windows. I synced a folder containing directory junctions from Windows to NAS, and the directory junctions weren’t copied to NAS, they we simply ignored (which is good, under the current state of things). So, maybe your statement is true for Linux, but not for Windows.

15

Yeah you’re correct, the docs state this:

Symbolic Links (synced, except on Windows, but never followed.)

16

I’d say that Syncthing is tuned for Linux, and other platforms are treated with lower priority. All features specific for Linux work fine, but on Windows or Android there are restrictions.

17

Hmmm. Yes, but I would not call it Syncthing’s fault that Windows and Android didn’t have a clear line of feature introduction and maintenance regarding symlinks. I mean, on linux symlinks were “the same” technical thing from the beginning. Windows did introduce them “hidden” with xp/ntfs and changed their behaviour and interpretation over time, recently switching over to some reparse points as an addition to their borked symlink handling.

2 Likes
18

I’m not saying whose fault it is, it’s just an observed fact. For instance, Syncthing cannot write SDcards on Android, and - as far as I learned - it’s fault of the GO language, as Android is not a primary platform for them. But if Android was a primary platform for Syncthing, there are possible workarounds, though they require a lot of work.

Concerning Windows, I’d say that the main usage of directory junctions is the same since Windows Vista - look how they are used inside user profile and system directories. Of course, there was a technical evolution, also there are several types of symlinks in NTFS, but the basic usage is IMO still the same.

19

So, I’ve took some time to play around with my little go knowledge and the Syncthing code. Got it to follow symlinks during the scan.

Source: https://github.com/Catfriend1/syncthing/pull/5/files

It’s correctly stating the folders and files beyond symlinks on Windows and Linux. To make it functional, TraverseSymlink checks have to be disabled, too, like in https://github.com/Catfriend1/syncthing/commit/441ef5cedf5ec727475b628f1f2a617089817d88 .

I don’t feel stable with the hack, but will do manual testing if it works because I’m curious.

1 Like
20

I’ve made a DRAFT implementation using an advanced folder option to “follow symlinks”. Please bear in mind that I’m not familiar with go. So questions like “why is it like this, could be more efficient” would be welcomed tips for me.

I cannot do the “risk management” in this concept I’d like to - at least - show to you. But I think it’s not very risky if one side decides to set the “follow symlinks” folder option, preferably together with a sendOnly folder.

I’m starting with node A in “normal Syncthing mode” where symlinks are ignored on Windows.

image

For the folder “symtest” on A I get:

image image

I’m connecting node B and sharing the folder with him: So B gets…

image

Now I’m turning my newly implemented option “follow symlinks” on for the folder “symtest” on A and make it sendOnly - for safety reasons - but could also have chosen sendReceive.

image image

Node B is now receiving everything “behind” the symlink “3” correctly.

image image

If I revert the option “follow symlinks” on A back to off. B will get the folder “3” removed again and the state of both nodes falls back to 16 (initial state from the beginning).

Again, I hope for the help to get this somehow clean to Syncthing. From my “data security” view, it’s nothing else you can burn your fingers like for example nasty ignoreDelete’s and ignorePermission’ settings users could set in a multi node cluster and then make some ignores versus unignores until they loose their data.

My implementation is not best from view of coding, sure, but please consider it as a sensible approach to make symlink’s possible in Syncthing on both Windows and Linux if admin users really need them. If unexperienced users would use this implementation, they would a) not know they could set “follow symlinks” and b) the one experienced user having reasons could well use it to help.

This would ease my syncs a lot as I’ve got a lot of folders with no common root and the ZFS filesystem which does not support mount --bind. Having copy jobs preparing a duplicate dataset “just for syncthing” is not nice but bearable at the moment. I’d be happy to help here.

Source: