Issue with Replication - /usr/local/bin/

Hello - and thank you for your help (in advance). I didn’t see anything like this, I apologize if this is a duplicate request.

We have an ignition application where “screens/files” reside in /usr/local/bin/ignition/sub/folders. I am trying to use syncthing to coordinate the replication and then from a master system… do a schedule push to github for source control. However, forget the github situation.

Rights for /usr/local/bin are normal linux rights (owned / group = root). However, ignition/sub/folders are all owned and grouped by our standard user/group. This is recursive and permissions are 755 across the board for the folders. I have processes for github copying objects in with these permissions without issues. I’ve used lsyncd with these folders without issue… however, lsyncd doesn’t fit the bill for the end goal. The SAME user/group is used across the board for these objects files for those processes as well. My guess is it has something to do with /usr/local/bin … but not sure why the permissions of ignition and beyond aren’t being honored when the other applications honor them.

With syncthing, something is keeping it from creating the files/folders on my test system. I keep getting (actual folder information removed): 2021-07-14 18:31:32 Puller (folder “stuff” (folderid), item “com/win/Sec”): syncing: creating parent dir: mkdir /usr/local/bin/ignition/sub/folders/stuff/com: read-only file system

For sure, the objects should be able to be created. I have the system.d service running as the mentioned standard user → systemd file below (sudo vi /etc/systemd/system/syncthing@.service):

[Unit]
Description=Syncthing - Open Source Continuous File Synchronization for %I
Documentation=man:syncthing(1)

After=network.target

[Service]
User=%i
ExecStart=/usr/local/bin/syncthing -no-browser -no-restart -logflags=0
Restart=on-failure
SuccessExitStatus=3 4
RestartForceExitStatus=3 4

ProtectSystem=full
PrivateTmp=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
~

I Started the service with: sudo systemctl start syncthing@user.service sudo systemctl enable syncthing@user.service

I Can sync using other folders without issue (ie /home/user)… but I really need these folders to sync between systems at the above mentioned folders due to the application. I need permissions to stay the same during replication as well. lsyncd worked great for all the above… but I need all changes on all systems to be replicated to and from each other. I hope I can get syncthing to work. FYI… moving folders is not an option.

thank you for your help.

Have you tried running syncthing without systemd? I wonder if the systemd protection is getting in the way.

Syncthing should run as the user that you want to own the files, which is not clear to me if that is happening now.

It’s also not obvious at what folder you are pointing syncthing.

I have not tried running without systemd. How would you run syncthing on centos 7 without systemd? (need it persistent on boot…etc). Lsyncd was also ran with systemd.

Syncthing IS running as the user (not root):

user 2947     1  0 17:26 ?        00:00:00 /usr/local/bin/syncthing -no-browser -no-restart -logflags=0
user  2996  2947  0 17:26 ?        00:00:13 /usr/local/bin/syncthing -no-browser -no-restart -logflags=0

The folder is (will be a total of 3 folders in this structure):

/usr/local/bin/ignition/sub/folder/folder1
/usr/local/bin/ignition/sub/folder/folder2
/usr/local/bin/ignition/sub/folder/folder3

for now… I am only trying to get folder 1 done and working. We only need these 3 folders for now, but may need to replicate more in the future. Where folder1/2/3 resides… there are other folders that reside there as well that will not be apart of syncthing replication.

I’ll try to help clarify anything necessary if more clarification is required. Thank you for the look and help.

So given the error is “read only filesystem” rather than a specific permission error, I am pretty sure it will be systemd protection that is getting in the way.

I’m no expert in systemd, but you should read about what each of the config parameters does, or try removing them.

Or try asking the question on some systemd mailing list.

It is systemd. Good call… it was the hardening settings.

For those that are interested: https://www.freedesktop.org/software/systemd/man/systemd.exec.html

  • search for the ProtectSystem section.
  • for full, it makes /usr, /etc, /boot, and /efi read only.

I need to figure out the overall impact of removing this for the service in this respect.

Thank you for looking and helping.

If you want to add addtional directories not allowed by default, you can do this:

sudo systemctl edit syncthing@user.service

(This will create and open an override file where you can put additional options, without modifying the original service file, which allows the service file to get auto-upgraded in the future, if it is managed by a package manager)

In the override file, use e.g the BindPaths option to add an addtional directory:

[Service]
BindPaths=/usr/local/bin/ignition

Save the file, restart the service (maybe reload the daemon before) and your syncthing should have read & write access to your folders, without removing any hardening options.

By the way, the service file you posted is already out of date: This is the current version: syncthing/syncthing@.service at main · syncthing/syncthing · GitHub

Thank you for the advice. I attempted to use the BindPaths, but I Couldn’t get /usr/local/bin/ignition to become read-able after adding the code to the systemd file. You do have to do a systemctl daemon-reload afterwards. I read up a bit on the usage, but couldn’t get it to work. I did try as your above example thinking “hey maybe it’ll work.” It didn’t work for me. I had to comment out the ProtectSystem=full again to get it working.

I’m all for the idea… and that would be ideal. It appears that this should be enough, BindPaths=/usr/local/bin/ignition as the other options are optional.

According to documentation:

Each definition consists of a colon-separated triple of source path, destination path and option string, where the latter two are optional. If only a source path is specified the source and destination is taken to be the same. The option string may be either “rbind” or “norbind” for configuring a recursive or non-recursive bind mount. If the destination path is omitted, the option string must be omitted too. Each bind mount definition may be prefixed with “-”, in which case it will be ignored when its source path does not exist.