The problem is that you must have consensus between the peers on whether the flag is on or off.
Someone can send you a message saying “I have a symlink foo pointing to /etc/passwd”, now next time you scan, that symlink will turn into a file because you have follow enabled, which you send back to the attacker.
Alternatively, you can “drop” symlink updates sent by remote peers if you have follow symlinks enabled, but then what does that mean? You are permanently out of sync? What happens when you disable the flag? Do you delete the files? Where do you get the history for that thing that you dropped?
There are a bunch of edge cases.
It’s not as trivial as just adding a flag, which you folks seem to be portraying.
If that was a response to my post, I understand that. But if on Windows symlinks are never treated as symlinks by ST - which seems to be the current state - and if it stays this way, then there is nothing to be worried about. I understand this might be a problem in Linux → Linux sync, but we are discussing the situation when one of the peers is Windows.
Windows symlink → Linux directory
Linux symlink → Windows directory
Windows symlink → Windows directory
In the last two cases, if the symlink was already created in the target (e.g. manually) before the sync, it can be followed even when writing.
The current state is
Windows symlink → Linux nothing
Linux symlink → Windows (empty) directory
Windows symlink → Windows haven’t checked, but expect empty directory
Sure, but this becomes a platform inconsistency, and I don’t think we want that.
Now it’s just lack of feature support for Windows, where as this would literally become difference in behaviour.
AFAIK, there are three types of symlinks in NTFS:
directory junctions, directory symlinks and file symlinks.
Directory junctions are used as “mount” for folders from another location, while directory and file symlinks seem to correspond more to the Linux symlinks. This distinction is underlined by the fact that directory junctions can be created by anyone, while symlinks can be created only by admins.
I think that ST should follow this platform specificity. Hence it might seem a good idea to treat directory junctions as directories, while symlinks could be treated like Linux symlinks. The only problem is that ST will never be able to create symlinks, as their creation requires admin elevation of rights.
I fact, I’m interested only in directory junctions. I understand that allowing symlink transparency may cause security problems, but I do not need symlink transparency.
Edit: I haven’t read any Microsoft document that describes purpose and distinction between junctions and symlinks. I describe only how they are actually used.
Edit2: What I have described corresponds also to how Windows File Explorer treats the distinction between directory junctions and symlinks, see https://stackoverflow.com/a/56973204/1259360.
Edit3: Transparency of symlinks (as opposed to transparency of directory junctions) might cause problems also because the target may be disconnected. Symlinks may point to network disks, which may be available only when the user is connected to VPN etc., and this would cause problems for ST. On the other hand, directory junctions may point only to local system disks. Disconnected for directory junctions means broken.
Refusing incoming updates with symlinks helps with the follow-symlinks case, sure. Audrius already talked about the effects of doing that. Trying to enforce it cluster wide is probably a no-go, though. Just dropping the connection probably isn’t a good idea either as it’s difficult for the user to understand and fix the situation.
Don’t underestimate the number of people enabling something just to test or because it sounds interesting. I do that all the time.
@Catfriend1 You are now testing symlinks on Linux, not on Windows, if I’m not mistaken? This diverts from the original and much simpler question, how to do that in Windows. Allowing transparency for directory junctions brings no security problems, because directory junctions are sort of like Linux mounts. But allowing transparency for symlinks (both Linux and Windows symlinks) brings difficult security problems, that you may or may not be able to resolve. I’m affraid that the directory junctions transparency will be burried with the symlink transparency, if you won’t succeed.
You keep insisting that they are different, but I believe they are exactly the same from Go’s API perspective, so I don’t think we could do anything even if we wanted.
Also, I don’t see how their behaviour is any different from symlinks, they effectively do the same thing.
I’ve already written several arguments in my previous posts. But try to search for “directory junctions vs symlinks”. First result gives me another argument:
The main difference is that, if you are looking at a remote server, junctions are processed at the server and directory symbolic links are processed at the client.
This argument also implies that it makes no sense in synchronizing directory junctions as “raw” objects, the same way symlinks are treated.
After a short search I found this link where they distinguish between juctions and symlinks.
Don’t comment only on my citations, just do the search yourself. There are plenty of resources that explain the difference. Technically, they may be similar, but conceptually they are different.
They import “internal/syscall/windows”. I don’t know golang, so I cannot tell whether this import is available to you, or whether there is another package that allows you to make the distinction.
You cannot convert everything in various platforms to Linux equivalents, because in many cases there are no Linux equivalents for the feature.
For instance, on Android, the application has to ask for various privileges first - you don’t need to do that on Linux, hence this as a platform specific feature. Or, you have to take into account in the code that there are colons and back-slashes in Windows paths - of course, you can convert back-slashes to slashes used by Linux, but you cannot get rid easily of colons, nor you can ignore them.
Or, have you noticed how bad turned out the Microsoft attempt to unify desktop and touch-screen applications? They ignored the difference between the platforms, and the result is that the “Metro” applications are hardly usable on desktop. They even changed the direction of scrolling with the mouse just to make easier to ignore the difference between mouse and touch-screen. But mouse is not a touch-screen. If Windows was the primary platform for your application, would you like to use its “metro”-style design on Linux? Of course not, on Linux you want the application to look and feel Linux-like.
Hence, it’s inevitable for your application to contain pieces of code specific for each platform you want to support.
There is no direct equivalent for directory junctions on Linux, they are something in between Linux mounts and symlinks. When you treat directory junctions the same way as symlinks, i.e. when you ignore the difference, you’re breaking the semantics.
I’ll not go into any technical points, simply because I don’t know enough about windows junctions. Instead I’ll take about communication that actually works towards getting what you want (mainly addressed @xarx, but I believe generally relevant).
There’s two ways (maybe more) to get a change in Syncthing (and probably most volunteer led open source projects): 1. DIY or 2. convince a contributor to do it for you. 1. is a much easier option from a communicative standpoint, but I get the impressions that’s not your focus. For 2. you need to remember that you try to get someone to do unpaid work, i.e. you need to provide enough motivation. And that bar is set by those that do the work, not you. If you state your use-case and provide some technical info, and the reaction is: Well, but what about that? Your reaction cannot be: Go read that up. You have lost at that point. The person that you want to do the work will obviously walk away. You provide all the requested information until they agree - or just say they not interested. That’s an inherent risk that you need to be prepared to take, and accept: They don’t even need superiour arguments: “Not interested” is a perfectly fine reason - it’s their time.
If you aren’t motivated to answer questions and put effort in convincing others why your idea is great and how it can work, there’s not point in going on at all, you’re just wasting your time. If you are motivated, read closely and address any issues brought up. Or at least acknowledge them if you can’t offer a solution or disagree.
I have not a problem to do the work myself, actually @Catfriend1 did most of the work. I can try to resolve the remaining problems, though I’ll probably need your assistance with that, as I have no knowledge of golang. Together we may be able to create a working branch that implements the feature.
But the problem is/was, that @calmh stated at the beginning very clearly that he’s against the feature to be incorporated into the main Syncthing branch. I’m losing interest if that means that I would have to maintain my own branch, if there’s no perspective for it to be merged into the main branch.
I’ve got the feeling that you all are Linux-oriented guys. I was trying to explain to you that your view of directory junctions from the Linux perspective is wrong. I provided several arguments and provided several links. I’ve got nothing against @AudriusButkevicius, but if he clearly didn’t bother to open them and read the information, it’s hard to persuade him. You must take into account that I’m not an expert in directory junctions, so I’m maybe unable to lay down the right arguments that would persuade you. But even if I was and gave you the right arguments, this wouldn’t help me to persuade you if you didn’t read them.
To sum that up - If you agree to incorporate the feature (when it is working) into the main branch, I’ll start investigating further and try to implement it, probably based on what @Catfriend1 already did. If you still need to be persuaded whether directory junction transparency is a good thing to do or whether directory junctions are really that different from ordinary symlinks, then ask me, or read the resources I offered to you. Currently I have no idea what more can I say without repeating what I’ve already written.
One problem is you are clearly focus on these junctions only, while that’s not the case for this thread and Catfriend1’s work. And you naturally got answers in that context, which you then discard because you feel they are besides the point. They are not. If you want to discuss something about directory junctions, open your own thread.
That’s cool!
As to the work done: Maybe true for your use case about junctions, no clue about that so far. However for following symlinks in general you clearly don’t get the point: Implementing following symlinks is not hard. And the last code I saw just did that, nothing more. There have been tons of concrete issues brought up in this thread and Catfriend1 took some steps to address it, but there’s still lots of stuff that isn’t fixed on a conceptual level.
What he stated was, that this has been brought up and attempted multiple times. He has no personal interest in it, i.e. won’t do it himself. On those two points I am certain I don’t misrepresent me, and I believe he would also agree to the following: Whether or not any solution to follow symlinks will get into Syncthing depends on whether all the concerns are addressed and the cost of mitigating these concerns and maintaining those long-term matches the value it adds. So far most concerns haven’t even been acknowledged, so yes at this point there’s no chance of getting this into Syncthing - that can change though.
See the first paragraph. And while people might be linux-centric, Syncthing is designed to be as system agnostic as possible. You don’t need to show something works “like on linux”, but how it can be integrated such that it makes sense everywhere or can be “safely ignored” on incompatible systems (no inconsistencies).
My proposal is that you write up your use case about junctions and your vision how it could work and post that to your own thread.
On a side-note: Without (naked) external links. If you can’t be bothered to cite relevant parts or paraphrase, I don’t think it’s fair to expect anyone else to be bothered with investing time into your proposal.
@imsodin, it was mine thread from the beginning, I was the one who started it, the directory junctions were discussed from the beginning too, see the thread title. My fault was only in that I didn’t see the distinction between directory junctions and symlinks, then. While I was focusing on directory junctions, from my Windows perspective, you were focusing on symlinks, from your Linux perspective.
I’m interpeting this that as @calmh was explicitly against that feature, so you are wrong even in this point. Though I agree with him when considering symlinks, I tried to persuade him and the others that junctions are different.
Yes, I agree with @calmh and think, that transparency of symlinks is a bad idea and you won’t be able to resolve all the security issues it brings. The key reason for that is in that they are interpreted at the client side. Though I wish you success in trying to overcome the security problems, I believe you won’t succeed, and that’s why I wrote
The rest of your post is based on false presumption that I hitchhiked your thread. No, you hitchiked mine.
Edit: I forgot to respond to what you write in point 3.:
I believe that the only way how you can make a truly “system agnostic” program is to focus on the features that are common for all of them, and ignore the rest. Which is obviously not the right way to follow, because you’ll end up with something similar to Microsoft “Metro” applications that are hardly usable elsewere than on Microsoft tablets - because they ignore specific features and posibilities of individual platforms, the desktop platform in particular.
I sincerely apologise for that - I got that obviously mixed up completely.
That I didn’t get until now, good that it’s out of the way.
Looking at the rest of his reply this statement was in the context of symlinks.
As far as I understand from your request and this golang issue turned up in a quick search it is the behaviour of golang to treat directory junctions as symbolic links. So now the question is do we want to treat junctions like directories instead of symlinks and if the argument applies generally (outside of Syncthing), lets see what upstream/golang thinks about it, and if not, is it feasible to implement in our filesystem abstraction (more like feasible without jumping through too many hoops).
An answer in the post you linked argued that junctions aren’t conceptually distinct from symlinks (apart from the implementation distinction):
NTFS Junctions and NTFS Symbolic links are really doing the same thing in the same way (reparse points), aside from the aforementioned differences in how they’re processed. In fact, technically, a Junction is a “symbolic link” in the more general sense of the word, and sometimes documentation might call a Junction a symbolic link, as is the case here. In such cases, “symbolic link” does not mean NTFS Symbolic Link which is different than a junction (see below).
This conflicts with the answer you linked to, that portrays junctions more like bind --mounts in linux (i.e. appear to the user as the directory it points at).
Yes, that’s the question. If we agree that it’s a good idea to treat directory junctions as directories, I’ll try to investigate if there’s a way how to do that; until then I’ll do nothing in this regard.
The golang issue you found I had seen too. It concerns the filepath.Walk method that has no protection against infinite recursion. It is a valid concern, and they resolved it by treating junctions the same way as symlinks, because that’s the “symlink side” of directory junctions that caused the problems. Actually, they had not many options to choose from. But that doesn’t mean that it is the same think. In other situations - and I believe this is the case of Syncthing too - the “mount side” of directory junctions is more important.
The answer you quote is completely correct. Technically, directory junctions are nothing more than symbolic links. Technically. It is written this way even in your quotation. But the “aforementioned differences in how they’re processed” are what distinguishes them from ordinary symlinks.
Use cases:
For instance, when I want to move a directory to another disk, e.g. because I’m running out of space on the original disk, I use a directory junction. (On Linux, you would mount the directory, in this case.) When I want to make a shortcut to another directory, e.g. logs on a server, I use a symlink.
When I want to backup a directory that contains a subdirectory redirected to another disk, I want to backup it completely, including the part stored on another disk. On the other hand, when I backup a directory containing symlinks, I do not want to backup the server logs to which they point.
The use-cases are completely compelling (on linux that would be mounts vs symlinks).
The difference about only absolute and local paths are possible seem like an implementation detail irrelevant in the context of syncthing (we anyway can’t follow anything remote like samba).
Other than that I didn’t yet find any characteristics or documentation that corroborate your distinction of the two. All the information I find says that junctions were there first, now symbolic links are getting introduced (well since XP, but not exposed to the user, not even now fully (requiring some admin config of windows to allow usage)). So my worry is that we are attributing meaning where there is none, which might lead to unexpected behaviour (differences).