Just a digression, but Dropbox in the past used to support junctions, and their notifications did not work for them either. The junctions would only get scanned and synced when Dropbox was being started (as there is no concept of periodic scans in it, at least as far as I know).
In comparison, Syncthing has more robust scanning mechanisms, so I believe that this should not be a big deal.
Ain’t nobody so big of a boy that they won’t come here to complain when it doesn’t work though. This is the thing with any new feature. It’s not just stirring together some code, it’s maintaining and supporting it in eternity. 
I am not aware of junctions having come up in the context of syncthing, except for what @xarx describes. And their use-case would be covered by the following, which is useful everywhere and doesn’t need dealing with junctions at all. Thus I would greatly prefer that.
Plus is anyone considering working on this and maintaining it after? I can see myself working on a multi-root folder, that might be useful for myself (this does not mean that I will do it). I certainly won’t do any special casing with safety-nets and so on for junctions. And I would probably not be in favor of a “big-boy” barebones version that can blow up in the users face, because users don’t respect dangerous settings and thus we get to support these blowups afterwards.
These all are various usages of directory junctions. Deliberately, I have omitted the use of directory junctions to collect multiple directories into a single shared folder.
So, what next? I can implement the feature, though it’ll take me some time. I’ll have to make the build process working, I’ll be fighting with Golang that I do not know, and I do not have much time. But I do want the feature, so I’ll make sure it’s implemented in a near future. Then I’ll send a pull request. Or do you want to implement that yourself?
Do you have any expectations concerning the notifications you were discussing just now? I don’t know how they work, so you would have to advice me.
In fact Dropbox says this:
Dropbox will follow Windows junction points and sync the files or folders they link to. However, any changes to those files or folders made from the Windows operating system will not sync again until the Dropbox desktop app is restarted. To get around this, move the original folder to your Dropbox and add a junction point from its previous location to link to its new location in the Dropbox folder.
They recommend inverting the relationship instead…
Are there junctions to files as well as directories, or are they just being unnecessarily general?
I have the directory junctions already set up on the disk, because of other reasons. I now only need Syncthing to support them. Multiple roots would work too, they might have a more general usage, but they would require to set up (and maintain) all folders again, just for Syncthing. Hence I prefer implementing the directory junctions transparency.
IMO, implementation of multiple roots would require much more work. And when it will be done? Is it in YOUR road plan?
Directory junctions are just for directories, not for files.
I’d say that this is a rather simple change, and it can be written easily maintainable.
I suspect that this isn’t a simple change, and that it will carry a significant maintenance and support cost just by it’s nature, as an oddity with relatively few users and with corner cases we’ve already established will cause disaster (recursive junctions) or non-working features (notifications). You can prove me wrong on the implementation part. Even then, a simple but half assed feature might be something we regret years later. (Compare for example “ignore deletes” which I implemented, was probably ten lines of code, we still have a support issues on it every week years later…)
The alternative (adding multiple folder roots within the same folder) might have none of these drawbacks and enable more interesting usages that we haven’t considered yet, while still solving your use case. That you might have to take a one time cost to reconfigure a folder to use the new feature doesn’t really enter into the calculations.
Is it on my roadmap? No.
Yes, but then they become just regular folders from the Dropbox’s point of view. One can do the same with Syncthing too. The real difference is that Dropbox does follow junctions located in its folder, while Syncthing does not.
I personally do not really need or care for this feature anyway, but just wanted to add some information from my long experience with Dropbox. I tried to experiment with junctions back then, but the lack of change detection (with no periodic scanner) ended up being a dealbreaker for me.
@Catfriend1 has already implemented that, mostly, though he did that for symlinks in general, not for directory junctions. I saw his changes, there were not many modifications. At the places where he tested for symlinks, I would call a general function “IsTraversable” instead. Implementation of this function would be encapsulated elsewhere, it’s not that difficult. The only difficulty would be only in tracking the traversed links in order to prevent infinite recursion. IMO nothing difficult and complicated.
If notifications wouldn’t work, would Syncthing stop synchronizing? As I understood in previous discussions, advanced features often lack some functionality, so this drawback seems acceptable to me.
Thinking about that again, this is not a full-fledged replacement for directory junctions. What if you want to sync a directory that contains directory junctions inside. How would you glue up the the complete directory from individual synchronized pieces? This would help only in the simplest scenarios, like “collecting multiple directories into a single shared folder”.
Catfriend1 did something, but I don’t think it’s the right approach. If I were going to do this, I’d run with the premise that junctions aren’t symlinks and implement a hack in the fs layer. There, you can lie and say that a given junction is not in fact a junction, it’s just a normal directory. Things would then probably mostly work without butchering the symlink checks.
What happens with the notifications is another thing. Does it successfully set up notifications and everything looks fine but they just don’t work? That’s probably not acceptable. Do we throw an error and say that you can’t use notifications because there are junctions? Might be OK.
You are the ones that have to say how the feature is to be implemented. But I do think that implementing the feature by hacking the fs layer and lying there would cause many troubles in the future. In maintenance, in particular. Because that would obfuscate the purpose of the changes, and prevent developers from using the standard Golang packages, as IsDir there would return a different result than IsDir in your package (if there’s such a function in your package - this was just an example). Moreover, you still would have to make the modifications related to prevention of infinite recursion - and for that you would need to know whether the current fs object is a dir or symbolic link.
Sure, IsTraversable should be put into the fs or related package. If you want to hide the junctions from the main code, you would have to replace all uses to IsDir with IsTraversable, where appropriate. Without that you would make the code maintenance difficult.
I myself would put everywhere explicitly “if IsDir() || (IsLink() && IsTraversable()) then…”. Such an implementation would be simple, then. But again, you are the ones who says how this is to be implemented.
You misunderstand, we already have our own fs layer as an abstraction. Implementing it there correctly illustrates how we interpret the underlying real fs, and is a lot cleaner in my opinion that passing disable flags all over the place.
You mostly just need to override it enough to make the IsDir and IsSymlink returns mean what you want them to mean. Everywhere already uses this fs abstraction.
(This is also where is put the hypothetical union fs for folders.)
You might be able to just take lstat_regular.go and make an lstat_windows.go that does the relevant bit syscalls and bit twiddling. At least I’d start there and then see what still doesn’t work.
OK. But anyway
Well, but should I start? For me, this discussion is still in the level “If we wanted that, we would do that like this. But we are not decided, yet.”
Yes, that’s one issue. One thing you could do there, that would also help in some other cases, is a function to return a directory’s inode number (Unix) / file ID (Windows). We could then keep a stack of those while traversing directories and stop if we reach one we’ve already seen. This would also defeat circular bind mounts (where such are possible) and assist with future rename detection as well.
Yeah, why not. I think having this at the fs layer is much more promising. If you can simply make an override for Lstat for Windows that treats junctions as dirs and everything works, that’s a tiny patch so no maintenance issues and it’s all good.
The basics won’t handle circular junctions and I’m not yet decided whether that’s something we must detect or we just let it slide for now. Probably it’s not impossible to detect as above, and then we might as well do it. We only need to do this tracking while doing a filesystem walk, and that code is also in there in the same package.
Good point. I didn’t see a reason why to keep track of ordinary directories.
Btw the better place to start is probably to copy out Lstat() from basicfs.go into the unix and windows specializations and there have the windows version do the extra syscall etc. Then you pass the extra flags or whatnot to the basicFileInfo which already has a windows specialization in basicfs_fileinfo_windows.go that does hacks around symlinks… Drop the symlink bit there if you know it’s a junction, and that’s probably all you need to do.
This behaviour should be dependent on the current Advanced configuration of the current directory. Is fs the right place? I.e. is the current fs layer behaviour already determined by the Advanced configuration?
There are no configurable things in there today, no. Are we sure this needs an off switch? I mean, there are multiple options of creating one, if so. One would be to make this a separate filesystem type. Today we have “basic” (this) and “fake” (in-memory). “junction-following” could be a subtype of “basic”.
I do not need the switch. But there are other users too, and they may have a different opinion on that.
And no doubt they will voice them, and then we’ll know the reasons. I’m generally pro avoiding knobs and switches until they’re obviously needed. This is one of those things where I think I’m fine with just having the one behavior until the opposite is proven.