syncthing@root but still permission denied?

Utini · January 23, 2023, 3:52pm

Hello everyone, I am creating system & mysql backups with Directadmin. The folder & files are owner by root:root:

xr-x--- 2 root  root  4096 Jan 23 12:26 apache
drwxr-x--- 2 root  root  4096 Jan 23 12:26 bind
drwxr-xr-x 5 root  root  4096 Jan 23 12:25 custom
drwxr-x--- 2 root  root  4096 Jan 23 12:26 mysql

How ever, when trying to sync them (or as I do, just send them to my backup server) I get “permission denied” an all those folders and files. Syncthing is exexuted as systemctl start syncthing@root.service

Any ideas / suggestions?

calmh · January 23, 2023, 4:01pm

ps aux | grep syncthing? Actual, full error messages?

Utini · January 23, 2023, 4:20pm

ps aux | grep syncthing

root     20098  0.0  0.0 733156 21204 ?        Ssl  13:16   0:00 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
root     20106  0.1  0.1 805896 60504 ?        SNl  13:16   0:23 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
root     30513  0.0  0.0   6544   884 pts/1    S+   17:16   0:00 grep syncthing

Error message from web GUI of the backup server for each file is the same. Or do you need any error checks from the main server on which the files are hosted?

syncthing checking file to be replaced: lstat /folder/path/to/folder: permission denied

calmh · January 23, 2023, 7:29pm

I meant actual error message, not pretend, because there are paths that are protected by the system unit file hardening, for example. Without that, your guess is as good as any, but that’s one direction you can look in.

Utini · January 23, 2023, 8:28pm

Uhm journalctl syncthing@root.service doesn’t show any real error. Got a hit where to look? :S

gadget · January 23, 2023, 9:00pm

Although “root” is the user with the most access, it’s still possible to lock out even root.

First step is to log on as root and verify that a simple command such as ls -lR /path/to/mysql works without errors.

Utini · January 24, 2023, 6:31am

ls -lR apache

apache:
total 64
-rw-r--r-- 1 root root    91 Jan 23 12:26 apache.md5
-rw-r----- 1 root root 60841 Jan 23 12:26 apache.tar.gz

Utini · January 24, 2023, 8:50am

I could add a simple “chown” / “chmod” command to my cronjob to fix this but I more not sure which values to set?

So basically:

* 6 * * 5 chown -R root:root /home/backup/ && chmod 1234 /home/backup/ && curl -X POST -H X-API-Key:papikey http://127.0.0.1:1234/rest/db/scan?folder=folderid

gadget · January 24, 2023, 3:43pm

Which Linux distro are you using?

Utini · January 24, 2023, 4:02pm

Debian

gadget · January 24, 2023, 4:25pm

Okay, so likely no SELinux involved.

What is the path to the apache directory starting from the root of the filesystem? (Please do not redact any directory names.)
Is the directory on a local filesystem or part of some mounted network share?

Any additional details, no matter how small, would be very helpful because the problem you’re having isn’t a Syncthing issue, it’s an OS configuration issue.

Utini · January 24, 2023, 6:10pm

Hey again,

the path is: /home/admin/admin_backups/01-23-23/apache/ It is the local file system, not mounted or shared.

The system that is creating this backup is “Directadmin” webpanel. Wouldn’t the easiest way be to run chmod/chown for the whole admin_backups folder via a cronjob?

gadget · January 25, 2023, 7:18am

So far, given the available info, there shouldn’t be a read permissions problem for root.

That’s certainly an option, but setting a umask to allow Syncthing read access would be more reliable and efficient. A cron job that only ran once every 24 hours would mean that Syncthing could only successfully sync new backups once a day.

But before tinkering with umasks and cron jobs, find out why a process running with root privileges is having a read permissions problem. Keep things simple by manually starting Syncthing without systemd, scripts or any other wrappers – i.e., /usr/bin/syncthing at a shell prompt so you can watch the run-time log output.

Utini · January 25, 2023, 9:06am

Alright so this is the output so far (I edited out ip/port/id):

/usr/bin/syncthing serve --no-browser --no-restart
[start] 2023/01/25 09:39:46 INFO: syncthing v1.23.0 "Fermium Flea" (go1.19.4 linux-amd64) deb@build.syncthing.net 2023-01-02 03:45:30 UTC [noupgrade]
[YXYVM] 2023/01/25 09:39:46 INFO: My ID: x
[YXYVM] 2023/01/25 09:39:47 INFO: Single thread SHA256 performance is 418 MB/s using minio/sha256-simd (418 MB/s using crypto/sha256).
[YXYVM] 2023/01/25 09:39:47 INFO: Hashing performance is 356.75 MB/s
[YXYVM] 2023/01/25 09:39:47 INFO: Overall send rate is unlimited, receive rate is unlimited
[YXYVM] 2023/01/25 09:39:47 INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
[YXYVM] 2023/01/25 09:39:47 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[YXYVM] 2023/01/25 09:39:47 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[YXYVM] 2023/01/25 09:39:47 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[YXYVM] 2023/01/25 09:39:47 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port x
[YXYVM] 2023/01/25 09:39:47 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [x]:x
[YXYVM] 2023/01/25 09:39:47 INFO: Ready to synchronize "home" (x) (sendonly)
[YXYVM] 2023/01/25 09:39:47 INFO: TCP listener ([::]:x) starting
[YXYVM] 2023/01/25 09:39:47 INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
[YXYVM] 2023/01/25 09:39:47 INFO: GUI and API listening on 127.0.0.1:x
[YXYVM] 2023/01/25 09:39:47 INFO: Access the GUI via the following URL: http://127.0.0.1:x/
[YXYVM] 2023/01/25 09:39:47 INFO: My name is "x"
[YXYVM] 2023/01/25 09:39:47 INFO: Device x is "HTPC" at [dynamic]
[YXYVM] 2023/01/25 09:39:47 WARNING: Syncthing should not run as a privileged or system user. Please consider using a normal user account.
[YXYVM] 2023/01/25 09:39:50 INFO: Established secure connection to x at x:x-x:x/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[YXYVM] 2023/01/25 09:39:50 INFO: Device x client is "syncthing v1.23.0" named "HTPC" at x:x-x:x/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[YXYVM] 2023/01/25 09:39:53 INFO: Completed initial scan of sendonly folder "home" (x)
[YXYVM] 2023/01/25 09:39:57 INFO: Detected 0 NAT services
[YXYVM] 2023/01/25 09:41:36 INFO: Paused folder "home" (x) (sendonly)
[YXYVM] 2023/01/25 09:41:41 INFO: Ready to synchronize "home" (x) (sendonly)
[YXYVM] 2023/01/25 09:41:41 INFO: Unpaused folder "home" (x) (sendonly)
[YXYVM] 2023/01/25 09:41:44 INFO: Completed initial scan of sendonly folder "home" (x)
[YXYVM] 2023/01/25 09:42:20 INFO: Paused folder "home" (x) (sendonly)
[YXYVM] 2023/01/25 09:42:22 INFO: Pausing x
[YXYVM] 2023/01/25 09:42:22 INFO: Connection to x at x:x-x:x/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256 closed: device is paused
[YXYVM] 2023/01/25 09:42:27 INFO: Resuming x
[YXYVM] 2023/01/25 09:46:01 INFO: Ready to synchronize "home" (x) (sendonly)
[YXYVM] 2023/01/25 09:46:01 INFO: Unpaused folder "home" (x) (sendonly)
[YXYVM] 2023/01/25 09:46:04 INFO: Completed initial scan of sendonly folder "home" (x)
[YXYVM] 2023/01/25 09:50:04 INFO: Established secure connection to x at x:x-x:x/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
[YXYVM] 2023/01/25 09:50:04 INFO: Device x client is "syncthing v1.23.0" named "HTPC" at x:x-x:x/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
[YXYVM] 2023/01/25 09:52:04 INFO: Overriding global state on folder "home" (x)

Scanning the /home/ folder on my main-server works fine. I did a few pause/start as you can see in the output. But the syncting status to my backup-server is still at 95% (with the 5% being the files with permission problems). There is no error in the webgui for now, it just says “out of sync”. I guess the next step would be to delete the folder from syncthing and add it again (as in creating a new sync folder) while having the shell open?

gadget · January 25, 2023, 4:23pm

Another quick test on your “main-server” box. As root:

rsync -na /home /dev/shm

The command above doesn’t actually copy/transfer anything, but it does cause rsync to walk the entire directory tree for /home. If there are no read errors, there will be no output.

How are Syncthing’s permissions on your “backup-server”?

Utini · January 25, 2023, 4:40pm

Hi again and thanks for your help!

No output from rsync on my main-server. My backup-server is based on Archlinux. So I followed the arch wiki (Syncthing - ArchWiki) and I am starting it with:

systemctl enable syncthing.service --user

It is syncing all the files on a mounted harddrive. I have several other syncthing folders / devices that sync into the same “parent” folder (/mnt/Files/Syncthing/).

I just checked systemctl status syncthing.service --user on my backup-server and found this:

● syncthing.service - Syncthing - Open Source Continuous File Synchronization
     Loaded: loaded (/usr/lib/systemd/user/syncthing.service; enabled; preset: enabled)
     Active: active (running) since Sun 2023-01-22 15:12:01 CET; 3 days ago
       Docs: man:syncthing(1)
   Main PID: 640 (syncthing)
      Tasks: 26 (limit: 19036)
     Memory: 314.7M
        CPU: 1h 31min 45.124s
     CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/syncthing.service
             ├─640 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
             └─669 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0

Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Puller (folder "x-Home" (x), item "admin/admin_backups/01-23-23/mysql/da_roundcube.sql.md5"): syncing: checking parent dirs: lstat /mnt/Files/Syncthing/x-Server/home/admin/admin_backups/01-23-23/mysql: permission denied
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Puller (folder "x-Home" (x), item "admin/admin_backups/01-23-23/bind"): syncing: checking file to be replaced: lstat /mnt/Files/Syncthing/x-Server/home/admin/admin_backups/01-23-23/bind: permission denied
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Puller (folder "x-Home" (x), item "admin/admin_backups/01-23-23/custom/usr/local"): syncing: checking parent dirs: lstat /mnt/Files/Syncthing/x-Server/home/admin/admin_backups/01-23-23/custom: permission denied
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Puller (folder "x-Home" (x), item "admin/admin_backups/01-23-23/custom/usr/local/directadmin"): syncing: checking parent dirs: lstat /mnt/Files/Syncthing/x-Server/home/admin/admin_backups/01-23-23/custom: permission denied
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Puller (folder "x-Home" (x), item "admin/admin_backups/01-23-23/mysql/full-mysql.md5"): syncing: checking parent dirs: lstat /mnt/Files/Syncthing/x-Server/home/admin/admin_backups/01-23-23/mysql: permission denied
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Puller (folder "x-Home" (x), item "admin/admin_backups/01-23-23/custom/etc/hosts.tar.gz"): syncing: checking parent dirs: lstat /mnt/Files/Syncthing/x-Server/home/admin/admin_backups/01-23-23/custom: permission denied
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Puller (folder "x-Home" (x), item "admin/admin_backups/01-23-23/custom/var/mail.md5"): syncing: checking parent dirs: lstat /mnt/Files/Syncthing/x-Server/home/admin/admin_backups/01-23-23/custom: permission denied
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Puller (folder "x-Home" (x), item "admin/admin_backups/01-23-23/custom/etc/group.md5"): syncing: checking parent dirs: lstat /mnt/Files/Syncthing/x-Server/home/admin/admin_backups/01-23-23/custom: permission denied
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: "x-Home" (x): Failed to sync 104 items
Jan 25 17:30:23 HTPC syncthing[640]: [x] INFO: Folder "x-Home" (x) isn't making sync progress - retrying in 1h4m0s.

On my backup-server I executed the commands you recommended me previously to check on my main-server:

ls -lR /mnt/Files/Syncthing/main-Server/home/admin/admin_backups
/mnt/Files/Syncthing/main-Server/home/admin/admin_backups:
total 8
drw-r----- 2 username users 4096 Jan 23 13:04 01-23-23
drwx--x--x 2 username users 4096 Jan 23 13:04 week-4

All the missing folders should be synced/copied into the folder “01-23-23”. That folder was automatically created by syncthing on the first scan/sync.

With Dolphin I can not create a file inside that folder (permission denied). Nano via shell works though but the file isn’t visible in Dolphin then.

gadget · January 25, 2023, 5:42pm

Okay, so Syncthing is running as a regular user…

With Syncthing on your Arch Linux server running as a regular user, it’s not going to be able to preserve all of the file/directory ownerships from your Debian server so it’s not worthwhile to enable “Sync Ownership” or “Send Ownership” (disabled by default).

On your Arch Linux server, either give the Syncthing user (UID 1000 according to the systemctl output) ownership of the subdirectory /mnt/Files/Syncthing/main-Server/home/admin/admin_backups or put the Syncthing user into the same user group and grant read/write permissions to the group.

Utini · January 25, 2023, 7:13pm

But the syncthing user and the folder owner are the same user. I checked that already. And since syncthing initially created that folder, it has to be.

ls -lR outputs myusername as owners and users as group.

Would it be smarter to run syncthing as root on my backup-server anyway so that I can restore a back up later more easily since the permissions would be correct on the backup?

gadget · January 25, 2023, 9:49pm

It was difficult to tell from the output because important details were redacted.

But, looking closer over the output from an earlier post…

ls -lR /mnt/Files/Syncthing/main-Server/home/admin/admin_backups
/mnt/Files/Syncthing/main-Server/home/admin/admin_backups:
total 8
drw-r----- 2 username users 4096 Jan 23 13:04 01-23-23
drwx--x--x 2 username users 4096 Jan 23 13:04 week-4

Note the permissions – it’s 0640 on 01-23-23 – even a root user requires execute permissions in order to list the contents of a directory.

No, it’s generally best to run as few services as possible as root, so keeping Syncthing running as it is now on your Arch Linux server is ideal (I’d also recommend doing the same on your Debian server).

Syncthing can easily sync the file and directory permissions, so all that’s required for a restore is to update the ownership.

(As you already know, Syncthing isn’t a backup program. Check out Duplicacy if you’re interested.)

Utini · January 26, 2023, 6:13am

Aha! There we have it. Thanks to you I checked a quick guide on chmod.

chmod -R 740 01-23-23/

fixed it and the sync has started/completed. How ever, the next backup will be created in a new folder as well (e.g. 01-26-23) and so it will have the same issue. It means that I would still need to add a “post chmod 740 command” to my backup creation process but I will also report this finding to the directadmin support forum

Regarding syncthing as a backup tool: I basically use “Directadmin” to create the backup. Syncthing is then syncing the backup contents to my backup server. Additionally syncthing is syncing the contents of /home/ (so basically all users and their files) to my backup server, which is handy because else I would need twice the space of /home/ on my main-server (1x for actual /home/ data and 1x for /home/ backup).

Regarding file and directory permissions: So I should enable “Send Ownership” & “Send extended attributes”? I was worried due to the warning “This can have a significant performance impact”.

Thanks again for your help!!