I’m not sure where you’re going with this, so not sure how to respond. Is this an experiment for your personal edification? Those are great. Are you recommending someone use this or aiming for inclusion in Syncthing? If so, my first reaction is that this probably allows another device to read and write files anywhere on the system, so that’s not something I’d recommend.
3 Likes
Side track - you can take a snapshot and mount that snapshot where you like, then destroy it. Many advantages over copying.
1 Like
Can you please elaborate this statement, or at least link to another discussion here where this was explained? Because I don’t see that’s true. Otherwise the whole NTFS links mechanism would be a security breach in Windows, because your argument would most probably apply to many situations where they are used, not just Syncthing. Thanks.
We don’t create symlinks on Windows so from a Syncthing perspective this maybe isn’t relevant there.
The patch disables our “don’t follow symlinks” protections. In a default setup I can then create a symlink foo -> /home, wait for you to sync it. Next scan you’ll follow the symlink and announce all of /home for me to sync under the foo “directory”. (Before it crashes due to following the symlink again, recursively, probably.)
Following symlinks safely is not as simple as just disabling our protections. Please don’t just throw out patches and expect me to explain why they don’t work.
You’ve disabled the safety grip on the chainsaw so that it can run without anyone holding it. That doesn’t make it a robotic lawnmower, it just makes it a really unsafe chainsaw.
5 Likes
Ok hold on, as I wrote on PM with Simon I’ll do some more testing and see if solutions to the risks can be found. @xarx no offense, but taking my draft for “discuss to want the feature” does not make sense yet. I’m just experimenting and hope to find out something useful.
I’m also grateful to get comments where risks/problems lie so I could investigate that.
1 Like
OK. From my point of view, following symlinks is fundamentally incompatible with letting other devices influence symlinks. Given that, I guess a minimal read-only implementation would be
- Following symlinks can only ever happen on send-only folders
- We must not allow nesting folders (which we otherwise do allow)
- Changing the folder type must disable following symlinks
- So what happens with the data already synced out there? It must be marked as deleted, which might be unexpected.
- Handle circular and chained symlinks (
foo -> .,foo -> ..,foo -> bar -> foo,foo -> fooetc)- “Handle” by stopping the folder on detection? It’s not obvious what the user intended to happen here I think.
- Symlinks can be to files or directories or specials (/dev/random)
- Consider whether allowing symlinks within a folder are a good idea or not.
- Symlinks can be dangling, do we represent that somehow or is it a scan failure (stop folder)?
- If not a scan failure, what happens to remote data when a symlink suddenly becomes dangling?
- We use
.stfolderto detect against unmounted storage being interpreted as a full delete. Can we do something similar when following symlinks to other partitions?
Probably more corner cases, that’s just from the top of my head. Handling symlinks on a send-receive folder is much more complicated. You’d have to at minimum disable processing incoming changes that affect symlinks, handle the case where a symlink points to a file on a different partition and where we would put the temp file when syncing a change to that file, etc.
A safer variation might be one we’ve explored before, where symlinks to follow must be explicitly named in the folder config. That doesn’t invalidate any of the above checks, but makes it safer to use in practice because inadvertently adding a new symlinks somewhere to cause a circular reference won’t be an error that needs to stop the folder, etc. I think that experiment failed because I tried to make it read/write but maybe it might be viable as read-only.
2 Likes
I’d say that this scenario cannot be performed on Windows, because directory juctions seem to be completely ignored by Syncthing. When I create a directory juction, it is not propagated to a linux machine. I haven’t tried to sync Windows with Windows, hence I don’t know whether directory junctions propagate there (but you wrote that ST dosn’t write symlinks on Windows). If they don’t, then there is IMO no reason to distinguish directory junctions from normal directories. But, of course, there are also other types of symlinks in Windows, and I haven’t tested yet, how Syncthing handles them.
If directory junctions are completely ignored, as written above, there’s IMO no need for this restriction. I don’t know whether other types of symlinks are handled by ST differently and whether Golang is able to distinguish different types of Windows symlinks.
All in all, if Windows symlinks are never written (as symlinks; currently it seems they are ignored), then following them and writing them as directories should be safe, IMO.
Edit: Of course, you can create a recursive structure using symlinks on your Windows computer, or point them to sensitive directories. But this is your own responsibility and depends on your privileges.
Ok, I’ve followed your posts and will answer later in detail. Making this post now to sum up what I’ve found during tests:
Case A) Syncing from Linux SO to Windows SR
A.1) Without “follow symlinks” at Linux SO: The windows target writes the symlinked directory out as an empty folder.
A.2) With “follow symlinks” at Linux SO: The windows target writes the symlinked directories’ contents as expected.
Case B) Syncing from Windows SO to Linux SR
B.1) Without “follow symlinks” at Windows SO: The linux target writes the symlinked directory out as an empty folder.
B.2) With “follow symlinks” at Windows SO: The linux target writes the symlinked directories’ contents as expected.
EDIT: I’ll now try the “/etc/passwd” glitch…
The problem is that you must have consensus between the peers on whether the flag is on or off.
Someone can send you a message saying “I have a symlink foo pointing to /etc/passwd”, now next time you scan, that symlink will turn into a file because you have follow enabled, which you send back to the attacker.
Alternatively, you can “drop” symlink updates sent by remote peers if you have follow symlinks enabled, but then what does that mean? You are permanently out of sync? What happens when you disable the flag? Do you delete the files? Where do you get the history for that thing that you dropped?
There are a bunch of edge cases.
It’s not as trivial as just adding a flag, which you folks seem to be portraying.
3 Likes
If that was a response to my post, I understand that. But if on Windows symlinks are never treated as symlinks by ST - which seems to be the current state - and if it stays this way, then there is nothing to be worried about. I understand this might be a problem in Linux -> Linux sync, but we are discussing the situation when one of the peers is Windows.
- Windows symlink -> Linux directory
- Linux symlink -> Windows directory
- Windows symlink -> Windows directory
In the last two cases, if the symlink was already created in the target (e.g. manually) before the sync, it can be followed even when writing.
The current state is
- Windows symlink -> Linux nothing
- Linux symlink -> Windows (empty) directory
- Windows symlink -> Windows haven’t checked, but expect empty directory
Edit: for file symlinks similarly.
Sure, but this becomes a platform inconsistency, and I don’t think we want that. Now it’s just lack of feature support for Windows, where as this would literally become difference in behaviour.
AFAIK, there are three types of symlinks in NTFS: directory junctions, directory symlinks and file symlinks. Directory junctions are used as “mount” for folders from another location, while directory and file symlinks seem to correspond more to the Linux symlinks. This distinction is underlined by the fact that directory junctions can be created by anyone, while symlinks can be created only by admins.
I think that ST should follow this platform specificity. Hence it might seem a good idea to treat directory junctions as directories, while symlinks could be treated like Linux symlinks. The only problem is that ST will never be able to create symlinks, as their creation requires admin elevation of rights.
I fact, I’m interested only in directory junctions. I understand that allowing symlink transparency may cause security problems, but I do not need symlink transparency.
Edit: I haven’t read any Microsoft document that describes purpose and distinction between junctions and symlinks. I describe only how they are actually used.
Edit2: What I have described corresponds also to how Windows File Explorer treats the distinction between directory junctions and symlinks, see https://stackoverflow.com/a/56973204/1259360.
Edit3: Transparency of symlinks (as opposed to transparency of directory junctions) might cause problems also because the target may be disconnected. Symlinks may point to network disks, which may be available only when the user is connected to VPN etc., and this would cause problems for ST. On the other hand, directory junctions may point only to local system disks. Disconnected for directory junctions means broken.
Just for completeness: I’ve used the “sync symlinks as they are” feature of Syncthing for the first time and now I see the problem with it. If I have two linux machines syncing the same folder and the folder contains a symlink to /etc/debian_version on #1, it gets synced over as symlink pointing to the same location on #2. So, yes, I now understand that there would be a huge “migration” problem to tackle if the user of one node suddenly decides to follow symlinks while the others are not doing it.
This sounds like a problem similar to what we’ve discussed before regarding large blocks. Does one node “infect” another with large blocks enabled so all communicate on the same basis? Or does the user need to make sure the large blocks setting is the same for all nodes of the cluster? Luckily we’ve just made the behaviour largeBlocks=Yes the default and didn’t need to introduce “cluster wide policies”. Cluser wide “central” policies would be the opposite of what Syncthing nowadays is: Decentral sync.
Ok, I’ve got another idea now. What about:
-
I assume, a user who wants to follow symlinks does this because he has “a plan” to do so and in which extend. This makes him unlikely that he will toggle it at later point for this specific folder to “sync symlinks as they are”.
-
The user has set “follow symlinks” for a folder on his device A. He does only require this behaviour on one “server” device and not on other device, but is not limited to.
-
The user shares the folder with device B.
-
Device B accepts the sharing request and gets a hint (like the folder name) on the protocol level to also set “follow symlinks”. Even in case the device does not actively need it because it should just receive the “virtual topology” of the user.
-
Let’s say the security risk case occurs, because the user is on Linux and changes the folder to “not follow symlinks”. When device B connects with A, it sees this on protocol level (folder option and probably incoming symlink object). B will then pause this folder emitting an error “folder settings not compatible with A, contact A and set the same setting regarding symlinks on all nodes”.
In this case, it wouldn’t put device B at risk because a symlink pointing to /etc/passwd couldn’t be received as he’s still in “follow symlinks mode” where this is not permitted.
The same happens to device A. Going back to the initial setup case, A follows symlinks and B too. If B turns back to “treat symlinks as they are” and attempts to send an /etc/passwd symlink to A knowing he would follow it, A would emit a warning because of inconsistent use of the folder setting.
The warning could be handled in two ways by the user(s) owning devices A + B.
- Both go back to “treat symlinks as they are”.
- Both stay on “follow symlinks”.
Going super cautious, I could imagine even stricter rules:
-
If once set, the follow symlinks setting of a folderId cannot be changed anymore (like the path). Then we would have docs to explain “you - the user - need to remove and readd the folder making sure, you don’t have symlinks or contents in place that won’t fit after the re-sharing”. I expect only admins to make symlinks intentionally, so requiring reading docs and understanding the consequences of symlinks and syncing a structure containing them in general could be expected (not only when using syncthing to handle such a directory tree).
-
Regarding the migration problem “v1.4.2 —> some future symlink supporting version”: If we detect a folder of an “old” syncthing without the followSymlink protocol info, we only allow to add a new device for that folder with follow Symlinks=No
Refusing incoming updates with symlinks helps with the follow-symlinks case, sure. Audrius already talked about the effects of doing that. Trying to enforce it cluster wide is probably a no-go, though. Just dropping the connection probably isn’t a good idea either as it’s difficult for the user to understand and fix the situation.
Don’t underestimate the number of people enabling something just to test or because it sounds interesting. I do that all the time.
1 Like
@Catfriend1 You are now testing symlinks on Linux, not on Windows, if I’m not mistaken? This diverts from the original and much simpler question, how to do that in Windows. Allowing transparency for directory junctions brings no security problems, because directory junctions are sort of like Linux mounts. But allowing transparency for symlinks (both Linux and Windows symlinks) brings difficult security problems, that you may or may not be able to resolve. I’m affraid that the directory junctions transparency will be burried with the symlink transparency, if you won’t succeed.
You keep insisting that they are different, but I believe they are exactly the same from Go’s API perspective, so I don’t think we could do anything even if we wanted.
Also, I don’t see how their behaviour is any different from symlinks, they effectively do the same thing.
I’ve already written several arguments in my previous posts. But try to search for “directory junctions vs symlinks”. First result gives me another argument:
The main difference is that, if you are looking at a remote server, junctions are processed at the server and directory symbolic links are processed at the client.
This argument also implies that it makes no sense in synchronizing directory junctions as “raw” objects, the same way symlinks are treated.
After a short search I found this link where they distinguish between juctions and symlinks.
Sure, they handle it, but as a end user of Go, you can’t interact with these things differently.