This is just a gimmick that has any meaning in DFS/CIFS/SMB, on the local machine that sentence has no meaning and they behave the same way.
Don’t comment only on my citations, just do the search yourself. There are plenty of resources that explain the difference. Technically, they may be similar, but conceptually they are different.
They import “internal/syscall/windows”. I don’t know golang, so I cannot tell whether this import is available to you, or whether there is another package that allows you to make the distinction.
Allowing transparency for directory junctions brings no security problems, because directory junctions are sort of like Linux mounts.
I’m looking at Linux as well because calmh is right about the security problem. Imagine a windows client misbehaves and sends a “fake symlink” that does not exist on disk to linux and then someone sets to follow symlinks.
But I didn’t want to misinterpret your post, I understood you’re looking at Windows only and see the “symlink following” as a gap filler because Syncthing on Windows currently doesn’t do anything with symlinks (no create nor stat). Would we have security concerns if we define following symlinks on Windows only? They couldn’t get created through Syncthing remotely on a Windows machine. I guess there’s nothing at the protocol level to tell a Windows Syncthing “hey, create a symlink I’ve seen on linux and noted to my database”.
Treating Windows other than Linux will then provoke questions like “why does Syncthing as a cross platform tool behave different on that OS?”. Okay, we’ve already got some Android hacks which make the Android client little different from “normal” Linux. I’m not very experienced with DB or protocol stuff… I think I’ll try next if I can stop a folder if it detects a dangling Symlink on Windows.
You cannot convert everything in various platforms to Linux equivalents, because in many cases there are no Linux equivalents for the feature.
For instance, on Android, the application has to ask for various privileges first - you don’t need to do that on Linux, hence this as a platform specific feature. Or, you have to take into account in the code that there are colons and back-slashes in Windows paths - of course, you can convert back-slashes to slashes used by Linux, but you cannot get rid easily of colons, nor you can ignore them.
Or, have you noticed how bad turned out the Microsoft attempt to unify desktop and touch-screen applications? They ignored the difference between the platforms, and the result is that the “Metro” applications are hardly usable on desktop. They even changed the direction of scrolling with the mouse just to make easier to ignore the difference between mouse and touch-screen. But mouse is not a touch-screen. If Windows was the primary platform for your application, would you like to use its “metro”-style design on Linux? Of course not, on Linux you want the application to look and feel Linux-like.
Hence, it’s inevitable for your application to contain pieces of code specific for each platform you want to support.
There is no direct equivalent for directory junctions on Linux, they are something in between Linux mounts and symlinks. When you treat directory junctions the same way as symlinks, i.e. when you ignore the difference, you’re breaking the semantics.
1 Like
I’ll not go into any technical points, simply because I don’t know enough about windows junctions. Instead I’ll take about communication that actually works towards getting what you want (mainly addressed @xarx, but I believe generally relevant).
There’s two ways (maybe more) to get a change in Syncthing (and probably most volunteer led open source projects): 1. DIY or 2. convince a contributor to do it for you. 1. is a much easier option from a communicative standpoint, but I get the impressions that’s not your focus. For 2. you need to remember that you try to get someone to do unpaid work, i.e. you need to provide enough motivation. And that bar is set by those that do the work, not you. If you state your use-case and provide some technical info, and the reaction is: Well, but what about that? Your reaction cannot be: Go read that up. You have lost at that point. The person that you want to do the work will obviously walk away. You provide all the requested information until they agree - or just say they not interested. That’s an inherent risk that you need to be prepared to take, and accept: They don’t even need superiour arguments: “Not interested” is a perfectly fine reason - it’s their time.
If you aren’t motivated to answer questions and put effort in convincing others why your idea is great and how it can work, there’s not point in going on at all, you’re just wasting your time. If you are motivated, read closely and address any issues brought up. Or at least acknowledge them if you can’t offer a solution or disagree.
3 Likes
@imsodin There are several points mixed together.
- I have not a problem to do the work myself, actually @Catfriend1 did most of the work. I can try to resolve the remaining problems, though I’ll probably need your assistance with that, as I have no knowledge of golang. Together we may be able to create a working branch that implements the feature.
- But the problem is/was, that @calmh stated at the beginning very clearly that he’s against the feature to be incorporated into the main Syncthing branch. I’m losing interest if that means that I would have to maintain my own branch, if there’s no perspective for it to be merged into the main branch.
- I’ve got the feeling that you all are Linux-oriented guys. I was trying to explain to you that your view of directory junctions from the Linux perspective is wrong. I provided several arguments and provided several links. I’ve got nothing against @AudriusButkevicius, but if he clearly didn’t bother to open them and read the information, it’s hard to persuade him. You must take into account that I’m not an expert in directory junctions, so I’m maybe unable to lay down the right arguments that would persuade you. But even if I was and gave you the right arguments, this wouldn’t help me to persuade you if you didn’t read them.
To sum that up - If you agree to incorporate the feature (when it is working) into the main branch, I’ll start investigating further and try to implement it, probably based on what @Catfriend1 already did. If you still need to be persuaded whether directory junction transparency is a good thing to do or whether directory junctions are really that different from ordinary symlinks, then ask me, or read the resources I offered to you. Currently I have no idea what more can I say without repeating what I’ve already written.
One problem is you are clearly focus on these junctions only, while that’s not the case for this thread and Catfriend1’s work. And you naturally got answers in that context, which you then discard because you feel they are besides the point. They are not. If you want to discuss something about directory junctions, open your own thread.
-
That’s cool!
As to the work done: Maybe true for your use case about junctions, no clue about that so far. However for following symlinks in general you clearly don’t get the point: Implementing following symlinks is not hard. And the last code I saw just did that, nothing more. There have been tons of concrete issues brought up in this thread and Catfriend1 took some steps to address it, but there’s still lots of stuff that isn’t fixed on a conceptual level. -
What he stated was, that this has been brought up and attempted multiple times. He has no personal interest in it, i.e. won’t do it himself. On those two points I am certain I don’t misrepresent me, and I believe he would also agree to the following: Whether or not any solution to follow symlinks will get into Syncthing depends on whether all the concerns are addressed and the cost of mitigating these concerns and maintaining those long-term matches the value it adds. So far most concerns haven’t even been acknowledged, so yes at this point there’s no chance of getting this into Syncthing - that can change though.
-
See the first paragraph. And while people might be linux-centric, Syncthing is designed to be as system agnostic as possible. You don’t need to show something works “like on linux”, but how it can be integrated such that it makes sense everywhere or can be “safely ignored” on incompatible systems (no inconsistencies).
My proposal is that you write up your use case about junctions and your vision how it could work and post that to your own thread.
On a side-note: Without (naked) external links. If you can’t be bothered to cite relevant parts or paraphrase, I don’t think it’s fair to expect anyone else to be bothered with investing time into your proposal.
@imsodin, it was mine thread from the beginning, I was the one who started it, the directory junctions were discussed from the beginning too, see the thread title. My fault was only in that I didn’t see the distinction between directory junctions and symlinks, then. While I was focusing on directory junctions, from my Windows perspective, you were focusing on symlinks, from your Linux perspective.
I’m interpeting this that as @calmh was explicitly against that feature, so you are wrong even in this point. Though I agree with him when considering symlinks, I tried to persuade him and the others that junctions are different.
Yes, I agree with @calmh and think, that transparency of symlinks is a bad idea and you won’t be able to resolve all the security issues it brings. The key reason for that is in that they are interpreted at the client side. Though I wish you success in trying to overcome the security problems, I believe you won’t succeed, and that’s why I wrote
The rest of your post is based on false presumption that I hitchhiked your thread. No, you hitchiked mine.
Edit: I forgot to respond to what you write in point 3.:
I believe that the only way how you can make a truly “system agnostic” program is to focus on the features that are common for all of them, and ignore the rest. Which is obviously not the right way to follow, because you’ll end up with something similar to Microsoft “Metro” applications that are hardly usable elsewere than on Microsoft tablets - because they ignore specific features and posibilities of individual platforms, the desktop platform in particular.
I sincerely apologise for that - I got that obviously mixed up completely.
That I didn’t get until now, good that it’s out of the way.
Looking at the rest of his reply this statement was in the context of symlinks.
As far as I understand from your request and this golang issue turned up in a quick search it is the behaviour of golang to treat directory junctions as symbolic links. So now the question is do we want to treat junctions like directories instead of symlinks and if the argument applies generally (outside of Syncthing), lets see what upstream/golang thinks about it, and if not, is it feasible to implement in our filesystem abstraction (more like feasible without jumping through too many hoops).
An answer in the post you linked argued that junctions aren’t conceptually distinct from symlinks (apart from the implementation distinction):
NTFS Junctions and NTFS Symbolic links are really doing the same thing in the same way (reparse points), aside from the aforementioned differences in how they’re processed. In fact, technically, a Junction is a “symbolic link” in the more general sense of the word, and sometimes documentation might call a Junction a symbolic link, as is the case here. In such cases, “symbolic link” does not mean NTFS Symbolic Link which is different than a junction (see below).
This conflicts with the answer you linked to, that portrays junctions more like bind --mounts in linux (i.e. appear to the user as the directory it points at).
1 Like
Yes, that’s the question. If we agree that it’s a good idea to treat directory junctions as directories, I’ll try to investigate if there’s a way how to do that; until then I’ll do nothing in this regard.
The golang issue you found I had seen too. It concerns the filepath.Walk method that has no protection against infinite recursion. It is a valid concern, and they resolved it by treating junctions the same way as symlinks, because that’s the “symlink side” of directory junctions that caused the problems. Actually, they had not many options to choose from. But that doesn’t mean that it is the same think. In other situations - and I believe this is the case of Syncthing too - the “mount side” of directory junctions is more important.
The answer you quote is completely correct. Technically, directory junctions are nothing more than symbolic links. Technically. It is written this way even in your quotation. But the “aforementioned differences in how they’re processed” are what distinguishes them from ordinary symlinks.
Use cases:
For instance, when I want to move a directory to another disk, e.g. because I’m running out of space on the original disk, I use a directory junction. (On Linux, you would mount the directory, in this case.) When I want to make a shortcut to another directory, e.g. logs on a server, I use a symlink.
When I want to backup a directory that contains a subdirectory redirected to another disk, I want to backup it completely, including the part stored on another disk. On the other hand, when I backup a directory containing symlinks, I do not want to backup the server logs to which they point.
1 Like
The use-cases are completely compelling (on linux that would be mounts vs symlinks).
The difference about only absolute and local paths are possible seem like an implementation detail irrelevant in the context of syncthing (we anyway can’t follow anything remote like samba).
Other than that I didn’t yet find any characteristics or documentation that corroborate your distinction of the two. All the information I find says that junctions were there first, now symbolic links are getting introduced (well since XP, but not exposed to the user, not even now fully (requiring some admin config of windows to allow usage)). So my worry is that we are attributing meaning where there is none, which might lead to unexpected behaviour (differences).
Security hole if directory junctions are synchronized by Syncthing the same way as symlinks, on Windows:
Consider a computer A of an attacker and a computer B, to which a folder is synchronized from A by Syncthing. The target directory on B is exported as a shared folder, e.g. in order the attacker can check whether everything has been synchronized correctly.
Now, the attacker creates a directory junction and a symlink, both pointing to C:\Users. If Syncthing synchronizes both to B as directory junction/symlink, the directory junction can be used by the attacker to access the content of C:\Users, to which he has otherwise no remote access. The symlink causes no security problems.
This vulnerability is caused by the fact, that directory junctions are processed at the server side. When the attacker uses e.g. the File Explorer to access the shared folder on B, the File Explorer allows him to traverse the directory junction. On the other hand, the File Explorer (and any other means) refuses to traverse the symlink. Not only that, Syncthing probably won’t be able to write the symlink, because this requires elevated administrator account.
Hence, there are only two possibilities for Syncthing how to treat directory junctions: either ignore them (which is the current behaviour), or convert them to directories.
1 Like
Just a little status update from my side. I’d continue my experiments with the code regarding symlinks and after I’ve found a way to discard incoming symlinks on linux when we are in followSymlinks mode, I had tried to get this working for symlink’ed files too.
This is very hard to figure out, so you all were right with your posts about security concerns. For example, I got it working to follow symlinked files and dirs with 2 linux machines. Machine A created a symlink to /etc/debian_version and machine B received the file correctly. Then I changed the contents on A and they were synced over to B. When I changed the regular file on B, A refused to finish the file in the puller. After I figured out how to do that, I got my contents synced over to A. But a syncthing-conflict file was also created being a symlink itself. Experimenting further with the goal A writes the contents from the regular file of B back to the symlink target, I hit the panic “error tried to version a symlink”. Uffff.
It’s very hard to get it all clean so it “just works for testing” as a starting point to know which code parts would be affected. Not even speaking of how FAR it would be to get this stable. At least, I’m having fun learning go and understanding Syncthing :-).
Current code for file and dir symlinks:
2 Likes
It’s great that you are having fun learning Go, but please do not expect this to land upstream.
The right approach to tackle this to land upstream is to write up a design doc on how it will work and how will all of the edge cases be covered.
2 Likes
I have similar issue to @xarx with additional dimension - to synchronize several Win laptops to NAS. Again - each one of them has several places/dirs to sync (not necessarily the same - some for sharing, some for backup) so my first thought was to create one catalog per laptop (C:\sync) with symlinks/junctions to all the places which need to be synchronized. I really love syncting, having used it successfully to synchronize/backup my phone (but that’s just pictures and messages - three dirs). Now, the idea of having 200+ dirs to arrange (instead of 8 – one per laptop) is really nightmare. I am all for option for advanced users for some symlinks and/or junction handling on Windows.
In the meantime, @Catfriend1 can you please share your binary with symlinks enabled? I would gladly test drive it, but I am not quite comfortable with compiling go myself.
@Catfriend1 can you please share your binary with symlinks enabled? I would gladly test drive it, but I am not quite comfortable with compiling go myself.
Syncthing symlink modded: https://drive.google.com/file/d/1BYM2zW_hA9feqISXvi69WiQmFsryp7cR/view?usp=drivesdk (this is the same code which was released as v1.4.1 afterwards, will rebuild soon)
I could, but please be aware it could have negative consequences in theory. I’m using the modification since end of 2016 in our company and never lost data (backups also in place!). It is advised to be very careful with the symlink mod. It’s mostly safe to run under Windows with making folders containing symlinks sendOnly. SendReceive works too, but other members may not delete folders that are symlinks on the sendreceive “host”.
The binary also proved protocol compatible to the official syncthing. It just announces directory symlinks as directories which official syncthing picks up on remotes.
Careful: do NOT create symlinks to files. I did not test this.
Okay, for users on Linux I’ve found a solution which makes creating a “virtual” directory tree for syncing possible by using:
- mount --bind
- Syncthing: Advanced folder option “markerName”
See [SOLVED] zfs pool, mount --bind: Syncthing deleting files from dirs after unmount
Basically, I’m just creating a virtual view directory tree like this:
/view/stfolderroot ^- Mount-binded-dir-pointing-elsewhere-1 ____^- .stsubfolder (directory) ^- Mount-binded-dir-pointing-elsewhere-2
Then I’m setting up a new folder in Syncthing at “/view/stfolderroot” and change “markerName” in the advanced options panel to “Mount-binded-dir-pointing-elsewhere-1/.stsubfolder”.
If the mount-bind target goes unavailable (“dangling”), Syncthing will stop the folder instead of picking up deleted content to its database.
This approach assumes the whole “/view/stfolderroot” contents are pointing to content on the same target disk, of course. If you have different disks, you should setup a virtual directory tree view for each disk to ensure it is properly stopped in case the disk or disk mount point goes unavailable. (due to ISCSI, fsck or sth else)
I’ve tested that I can distinguish directory junctions from ordinary symlinks in Golang (on Windows). Using syscall to a kernel32.dll function.
Does it suffice like this?:
Intent: Treat directory junctions on Windows as directories
There are two types of symbolic links in Windows NTFS - ordinary symlinks, and directory junctions. (They both are NTFS reparse points; other reparse point types are not considered as symbolic links). As discussed above, there is a substantial distinction between processing of symlinks and directory junctions - in particular, NTFS symlinks are similar to Linux symlinks, while directory junctions are something in between symlinks and Linux directory mounts. From the Syncthing perspective, the directory junctions can be safely treated as directories, in contrast symlinks can’t (this could cause security issues).
In Windows, directory junctions are used to mount directories from other parts of the filesystem. When there is a need to access a directory transparently (i.e. without knowing there’s a redirection) from within another directory, directory junction is the right way to do that. Windows itself uses directory junctions on tens of locations exactly for this purpose, while symlink is used only at one particular place.
Syncthing currently fails to traverse directory junctions ignoring this important Windows feature. Unfortunately, there’s no workaround for this in Windows (as opposed to Linux), which causes an almost insurmountable problem when one needs to backup many (tens) directories that are scattered on many various locations in the filesystem.
This is a proposal for implementing the directory junctions traversal in Syncthing:
-
Make traversal of directory junctions optional: Historically, the usage of directory junctions evolved, and even now not everyone may want the directory junctions to be traversed. Hence, I propose a switch in Advanced configuration that would enable or disable directory junctions traversal. The switch would be per folder, not global, enabling different folders to have a different setting.
-
When directory junctions traversal is enabled, directory junctions will be treated as ordinary directories, and Syncthing will synchronize them to other devices like ordinary directories. Upon write, Syncthing won’t create directory junctions, but if they are already present, it shall traverse them.
-
When directory junctions traversal is disabled, directory junctions will be synchronized as empty directories. (Another possibility would be to ignore them, but definitely NOT sync them as symlinks. I prefer treating them as empty directories.) Hence, when the switch allowing directory junctions traversal is turned off, the content of the directory will be synchronized as deleted. Upon write, directory junctions will fail to be traversed. (There’s no point in allowing to traversal while writing if the traversal is prohibited while reading. I haven’t checked (yet) how Syncthing currently resolves the situation when at target there’s a directory junction (or symlink), while at source there’s an ordinary directory - does the sync fail?)
-
Special case - invalid directory junction This can happen when the directory junction point to a removable disk (symlink should be used instead) or if the target directory was removed/renamed. In every case, this is an error condition, the directory junction is corrupted. In this case, either the directory junction should be treated as empty, or as unchanged. I prefer the second option, as it is fail safe, though I do not know whether this is possible in Syncthing. Of course, syncronized changes from other devices shall fail to be written to corrupted directory junctions.
-
Special case - infinite recursion Directory junctions can point to an ancestor directory, and cause infinite recursion this way. Such a situation can be either handled by Syncthing, or ignored (the user will be responsible not to allow that). Of course, it would be better to handle such a situation by Syncthing. This would be relatively easy to implement, but it would require to pass on a context object upon a directory walk/traversal. I haven’t checked whether a suitable context object is awailable in Syncthing code, or whether it would have to be introduced.
@AudriusButkevicius Is this what you asked for?
1 Like
Sounds workable. I mean, the fact that recursive junctions are allowed seems bizarre, but can be handled by tracking which junctions we’ve traversed and bailing if we come back to one we’ve seen before. There’ll be some corner cases to handle there as we sometimes do partial scans deep in a folder (not having traversed there to begin with) but there is a symlink check so I think it could be implemented there. We might also get stuff like filesystem notifications from in there, probably. Or if those present as the non-junctioned name then we need to handle that too. Oh, and if we use notifications and the junction gets removed we probably need to handle that somehow. Junctioning folders into each other might have some special cases as well, I haven’t fully thought that through.
What are the current uses of junctions in Windows where Syncthing might run into them? I mean, by default, not because of wanting to collect multiple directories into a single shared folder.
If that’s the only real use case, it might be an alternative to allow multiple folder roots in a folder, “mounted” at different places. There’s be trickiness there too, to be sure, but it might be more generally useful.
2 Likes
I think this should end up as a flag, which just treats junctions as directories, nothing else.
If there is recursion, let the application fill it’s database with trash and blow up in the users face. The user must be wearing big boy pants when deciding to enable it, hence should have checked that there are no cycles.
Junction pointing at something invalid handling will depend on what the error is when you try to “stat” it. If it returns “does not exist”, then we’ll go ahead and nuke all the data behind it, if it returns something else we’ll just assume it’s a permission error and leave it alone. Again, no special handling needed I think, and we should leave the existing logic to handle.
I think notifications will simply not work for this, but again, it’s a big boy feature, let the big boy handle it. I don’t think we’ll modify the watcher to crawl folders to check for junctions to setup recursive watches in the resolved locations.
1 Like