Hi there!!! I am needing some guidance on how to make extended attributes works. We have installed Syncthing on two different AlmaLinux servers as service users. It is syncing files over with the correct ownership and group one direction. When we turn on the bidirectional sync we were seeing chmod permission issues – we enabled send & sync extended attributes and we are still seeing chmod operation not permitted errors. We disabled SELinux on both servers and still not syncing.
To clarify it’s only the octal 2775 that seems to have trouble syncing.
The exact error is:
Puller (folder “Linux20 Project” (3ht5y-xcnkt), item “path/to/file/svg-27437.jpg”): syncing: shortcut file (setting permissions): chmod path/to-file/svg-27437.jpg: operation not permitted
Chmod doesn’t really have to do with either ownership or extended attributes, but if you sync ownership have you made sure to grant Syncthing the required capabilities?
It’s unusual to setgid and/or mark an image file executable, so should look into why that’s being done at all.
But as far as the chmod 2775 failing, check the mount options for the filesystem where Syncthing is writing to. Are you using the nosuid option? If so, setuid/setgid isn’t allowed.
Sorry to confuse you – those were two different issues…
We are having issues syncing sticky bits (octal 2775) across servers.
We were seeing error chmod errors on certain files - even after following the documentation regarding granting the service user the correct permissions.
The SELinux issue was fixed by disabling SELinux on the server.
Please note that chmod 2775 doesn’t set the sticky bit (there’s actually no octal value equivalent). Also, the sticky bit only applies to directories and are ignored for files.
What type of filesystem are the affected files being written to?
What is the output from the command stat /path/to/file/svg 27437.jpg and/or other affected file?
Shouldn’t normally have to disable SELinux, and ideally, simply adjusting the security context corrects the issue.
I think Im following what you are saying – It just seems like the octal 2775 directories are syncing over but the permissions aren’t updating on the second server.
For the filesystem when I ran the stat command this is what came back >>
Size: 3233 Blocks: 8 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 5634466 Links: 1
Access: (0664/-rw-rw-r–) Uid: ( 48/ apache) Gid: ( 1004/ project)
Access: 2019-03-06 06:42:33.000000000 -0600
Modify: 2019-03-06 06:42:33.000000000 -0600
Change: 2023-03-12 12:02:58.549290906 -0500
Birth: 2023-03-11 15:44:09.151898868 -0600
I think focusing on chmod 2775 has been an unfortunate red herring that’s leading you down the wrong diagnosis and solution.
First, because chmod 2775 is in no way related to the sticky bit. The 2 in chmod 2775 sets the set-group-ID bit, resulting in the following directory permissions:
drwxrwsr-x 2 user user 40 Mar 12 21:47 dir1
Same directory with the sticky bit enabled:
drwxrwxr-t 2 user user 40 Mar 12 21:47 dir1
Now combining both setgid and sticky bits:
drwxrwsr-t 2 user user 40 Mar 12 21:47 dir1
Using the preformatted text markup for easier reading…
According to the stat output above, the file in question is owned by apache:project (UID: 48, GID: 1004).
So, the most important question is, under which user is Syncthing running as on the second server?
If it isn’t apache:project or root:root (hopefully not the latter), then enabling syncing of extended attributes in Syncthing; disabling SELinux; and/or any other workarounds won’t help.
Your original post said:
If the first server is “A”, and the second server is “B”, given the facts so far, it sounds like B → A is syncing fine but A → B isn’t.
The second server isn’t set up the same way as the first server if the first server has been able to preserve the user and group without any errors.
Server A to Server B syncing did not show any errors after the sync was completed. After I enabled the bidirectional sync that is when the error “Out of Sync” appeared - upon investigating it appeared that the only errors in the logs at the time were files in directories that had an octal of 2775 – the other files/directories appeared to be in good shape.
Server A Syncthing User Groups:
syncthing : syncthing apache project
Server B Syncthing User Groups:
syncthing : project apache
From a usability standpoint, it wouldn’t work by itself. The user process still has to acquire elevated permissions via the “sudo” command. And unless that particular user also has no password, you’d have to automate answering the password prompt. (There are all kinds of ways to make it work, but long term it just adds unnecessary complexity to system management.)
Is Syncthing launching at server boot time? If so, how are you doing that?
Server A Passwd: syncthing:x:1003:1005::/home/syncthing:/bin/bash
Server A Syncthing User Group: syncthing:x:1005:apache,ejmccull
Server A Project Group: project:x:1004:apache,ejmccull,syncthing
Server B Passwd: syncthing:x:1004:1004::/home/syncthing:/bin/bash
Server B Syncthing User Group: syncthing:x:1002:apache,ejmccull
Server B Project Group: project:x:1004:apache,ejmccullough,syncthing
Including the project group for visibility as the files/directories we are dealing with are in the “project” group.
Syncthing is launching on boot I ran this command to do so on both servers & appears to run after the server is rebooted:
sudo systemctl enable syncthing@syncthing
Ah… it confirms my initial theory as to the sources of the problem.
UIDs and GIDs…
Let’s start with a little table comparing the user ID and group ID for user “syncthing” on each server:
User Server A Server B
---------------------------------
syncthing 1003:1005 1004:1004
Although it’s not mandatory that user “syncthing” have the same UID:GID pair on both servers, the only way that Syncthing on Server B can assign 1003:1005 to a file/directory that’s been synced from Server A is if user “syncthing” on Server B has elevated privileges (the same thing applies to the other direction).
By default, Unix/Linux will not allow a non-root user to arbitrarily assign a file/directory to another user because it opens the door to all kinds of issues (e.g., why worry about storage quotas if you can simply make a huge Blu-ray video file owned by another random user while still keeping it in your home directory?).
The other issue is with your “project” group ownership of files/directories…
On Server A, user “syncthing” is a member of the “syncthing” and “project” groups (primary and secondary groups, respectively).
But on Server B, user “syncthing” is only a member of the “project” group.
So, let’s walk through a hypothetical example. User “bob” has created a file on Server A that’s readable and writable by the “project” group. Output from the command ls -l:
-rw-rw-r-- 1 bob project 1 Jan 1 00:00 A.jpg
Great. Now let’s see what the UID:GID look like via ls -n:
-rw-rw-r-- 1 1111 1004 1 Jan 1 00:00 A.jpg
Since the “project” group’s GID is 1004, and user “syncthing” on both Server A and Server B are members of group 1004, there are no read/write/delete issues.
However, user “syncthing” doesn’t own file A.jpg, so changing the permissions and/or ownership isn’t allowed. Only a file’s owner – or a user with root privileges – can do that.
As long as the list of files/directories are owned by a mix of users (even though they are all members of the “project” group), you cannot enable Sync Ownership and must enable Ignore Permissions in Syncthing unless you run Syncthing with root privileges.
It’s not an issue with octal numbers for the chmod command, SELinux, Syncthing, and/or something else that hasn’t been considered yet.
Another option is to log on as user “syncthing” and use the following command:
systemctl --user enable syncthing@syncthing
(There are pros and cons to each technique. Search this forum for “systemctl” to see earlier posts about the same topic and usage tips.)
If the end goal is to preserve the original file/directory ownerships – i.e. a file owned by “bob” in the “projects” group is mirrored as-is between Server A and Server B, then matching UID:GID for the user “syncthing” won’t help. The only solution is to run Syncthing with elevated permissions.
But, if preserving the original file/directory ownership is optional, then there are several options available.
I posted the link for that in the first reply in the topic, up above. It’s not the user you give elevated privileges, it’s the Syncthing binary/process.
You can also try using our Docker container, which will grant Syncthing the appropriate privileges if you set the environment variable PCAP=cap_chown,cap_fowner+ep.