Syncthing Permission Denied even though user running process has access

Support
1

I’m running syncthing in a LXC on Proxmox 8.1 to sync my photos. The photos are on an external HDD, mounted on the host at /mnt/hoststorage/photos and passed through as a mount point to the container, monunted at /mnt/lxcstorage/photos. The filesystem throughout is ext4.

The photos are owned on the host by the user:group mediawriter:mediawriters with rwxrwxr-- permissions.

User ‘root’ in the container is mapped to be a member of the group ‘mediawriter’ on the host. I can access, read and write all the relevant files throguh the terminal. (e.g. mkdir /mnt/lxcstorage/photos/test)

Syncthing, however, cannot access the files with ‘permission denied.’ I have absolutely no idea why, since it is running as root and root has access to the files/folders. Any help would be much appreciated:

Logs and hopefully useful output are below:

Syncthing Log (trunkated somewhat since it’s much of the same)

YYYY-MM-DD HH:MM:53 Loading ignores: lstat /mnt/lxcstorage/Photos/User/UserPictures/.stignore: permission denied

YYYY-MM-DD HH:MM:53 Ready to synchronize “User Pictures” (xxxx-xxxx) (sendreceive)

YYYY-MM-DD HH:MM:53 Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting

YYYY-MM-DD HH:MM:53 Loading ignores: lstat /mnt/lxcstorage/Photos/User2/User2Pictures/.stignore: permission denied

YYYY-MM-DD HH:MM:53 Ready to synchronize “User2 Pictures” (xxxx-xxxx) (sendreceive)

YYYY-MM-DD HH:MM:53 Loading ignores: lstat /mnt/lxcstorage/Photos/User/UserVideo/.stignore: permission denied

YYYY-MM-DD HH:MM:53 Ready to synchronize “User2 Camera” (xxxx-xxxx) (sendreceive)

YYYY-MM-DD HH:MM:53 Ready to synchronize “User2 Videos” (xxxx-xxxx) (sendreceive)

YYYY-MM-DD HH:MM:53 QUIC listener ([::]:22000) starting

YYYY-MM-DD HH:MM:53 Failed initial scan of sendreceive folder “User Pictures” (xxxx-xxxx)

YYYY-MM-DD HH:MM:53 Error on folder “User Pictures” (xxxx-xxxx): stat /mnt/lxcstorage/Photos/User/UserPictures: permission denied

YYYY-MM-DD HH:MM:53 Failed initial scan of sendreceive folder “User2 Pictures” (xxxx-xxxx)

YYYY-MM-DD HH:MM:53 Error on folder “User2 Pictures” (xxxx-xxxx): stat /mnt/lxcstorage/Photos/User2/User2Pictures: permission denied

YYYY-MM-DD HH:MM:53 Failed initial scan of sendreceive folder “User2 Videos” (xxxx-xxxx)

YYYY-MM-DD HH:MM:53 Error on folder “User2 Videos” (xxxx-xxxx): stat /mnt/lxcstorage/Photos/User2/User2Videos: permission denied

Check owner of Photos dir on host system:

ls -l /mnt/hoststorage/

drwxrwxr-- 5 mediawriter mediawriters 4096 Mmm D hh:mm Photos

(have also checked anda all subdirectories share the same permissions)

Check owner of Photos dir on LXC:

ls -l /mnt/lxcstorage/

drwxrwxr-- 5 nobody mediawriters 4096 Mmm D hh:mm Photos

(I assume the issue might have something to do with the ‘nobody’ here, but have no idea why it would make a difference or how to fix it)

Check Syncthing is running as root (on LXC container)

ps aux | grep syncthing

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

root 161 0.0 0.8 1249208 16896 ? Ssl 16:06 0:00 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0 root 222 0.0 2.4 1251384 51788 ? SNl 16:06 0:05 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0 root 391 0.0 0.0 3324 1536 pts/1 S+ 16:47 0:00 grep syncthing

Check root is a member of the mediawriters group (on LXC container)

id

uid=0(root) gid=0(root) groups=0(root),1308(mediawriters)

Check that root can read/write the directory from LXC:

root@syncthing: mkdir /mnt/lxcstorage/Photos/test && ls

Other Photo Folders test

I’m not sure if there’s a specific command to check that the group is passed through correctly, but if reload the container with the passthrough commented out in the LXC .conf, the above command gives me permission denied.

Any help sorting this, gratefully received

2

Privileged or unprivileged container?

Is the container using a bind mount to access the mount point on the host?

Is the external HDD connected via USB?

3

Unpriviliged container (although making it priviliged doesn’t seem to make a difference)

Yes.

Yes.

Thank you!

4

It’s better for security if it’s unprivileged, so that’s fine.

The most visible difference is how UIDs and GIDs are mapped. In unprivileged mode, users in the container have UIDs and GIDs that are offsets from the (default) of 100000. So root in the container has UID 0, but to the host, it has the real UID 100000 (i.e. 100000 + 0). Likewise, if there’s a user in the container with UID 1234, it’s real UID is 101234 to the host.

With ext4 on the USB drive (if I understood correctly), I suspect it’s either an issue with the UID/GID mapping and/or ACLs.

If there are no other containers that require access, it’s simpler and more secure mounting the USB drive directly within the container. With bind mounts, there are restrictions and complications with UID/GID mapping (especially for unprivileged containers). With a direct mount, the container controls all of the permissions and the UID/GID in the container apply to the filesystem on the USB drive.

5

The most visible difference is how UIDs and GIDs are mapped. In unprivileged mode, users in the container have UIDs and GIDs that are offsets from the (default) of 100000. So root in the container has UID 0, but to the host, it has the real UID 100000 (i.e. 100000 + 0). Likewise, if there’s a user in the container with UID 1234, it’s real UID is 101234 to the host.

I have this set up to map across - and have full access to the files when I use the terminal.

If there are no other containers that require access, it’s simpler and more secure mounting the USB drive directly within the container.

Unfortunately, other containers do need access. I’m not having any issues with the other containers that are accessing the files; they all work fine and the UIDs are passed throguh in the exact same way. It’s only Syncthing (and only the syncthing application, since the root user is fine) that seems to be having problems :frowning:

6

In your original post, you included the following snippets:

Is the mediawriters GID 1308 on the PVE host?

If it is, root inside the container is a member of the mediawriters group with GID 1308, but that maps to 101308 on the PVE host.

7

Yes - mediawriters is 1308 on both PVE host and LXC container - with mapping between them in the CTID.conf.

in CTID.conf

lxc.idmap: g 0 100000 1308

lxc.idmap: g 1308 1308 1

lxc.idmap: g 1309 101309 64227

lxc.idmap: u 0 100000 65536

In /etc/subgid

root:1308:1

root:100000:65536

Host:

getent group mediawriters

mediawriters:x:1308:

Container

getent group mediawriters

mediawriters:x:1308:

The same mapping has been done successfully in other containers, and in this container allows the root user of the syncthing LXC to rwx in the applicable directory. (e.g. mkdir /lxcstorage/Photos/test succeeds). This just doesn’t seem to transfer over to syncthing itself (running as root).

8

Additionally, the logs in the Syncthing GUI say

stat /mnt/lxcstorage/Photos/somefolder: permission denied

However if I go into the LXC terminal and run

stat /mnt/lxcstorage/Photos/somefolder

This succeeds. The container root user does have the permission to do this.

Size: 4096 Blocks: 8 IO Block: 4096 directory

Device: 8,18 Inode: 85532677 Links: 10

Access: (0774/drwxrwxr–) Uid: (65534/ nobody) Gid: ( 1307/mediawriters) etc

It seems to be an issue somewhere that the syncthing application isn’t utilising all the existing permissions of its root user but… i’m stumped beyond that.

EDIT: it similarly gave

lstat /mnt/lxcstorage/Photos/somefolder/.stignore: permission denied

I’ve just gone into the terminal and specifically ensured that this file is owned by the syncthing root user. No dice.

10

On the host side, what does ps -efw | grep syncthing return when Syncthing is running inside the container?

11

100000 279788 279467 0 23:38 ? 00:00:00 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0

100000 279847 279788 8 23:38 ? 00:00:01 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0

root 280039 279699 0 23:38 pts/2 00:00:00 grep syncthing

Looks like it’s running on its mapped userid. Is it possible that passing through the group membership gives permissions when accessing the files through terminal, but not to the process?

I also tried to just add 100000 to the group mediawriters on the host to see what happened, but it said the user didn’t exist. I can’t run id or getent on it, either.

12

I’m still reviewing the info from the earlier post and thinking the process thru, but my current thoughts…

I can’t think of a reason why it’s specific to Syncthing. I suspect that other programs would also be affected.

One thing I noticed is that the GID is 1307 in the output from stat /mnt/lxcstorage/Photos/somefolder while your ID map specifies 1308.

13

One thing I noticed is that the GID is 1307 in the output from stat /mnt/lxcstorage/Photos/somefolder while your ID map specifies 1308.

Whoops, that’s my fault, sorry. While I was keeping testing between those posts I set up a new container and group mapped to 1307 just to see if it was something specific to that group/gid. (It wasn’t. Exact same issues). Must have copied the output from the wrong container.

Have since deleted the 1307 container and put everything back to 1308. Reran the commands just to check, and no dice - exactly the same output, just with 1308 instead. Strange.

Size: 4096 Blocks: 8 IO Block: 4096 directory

Device: 8,18 Inode: 85532677 Links: 10

Access: (0774/drwxrwxr–) Uid: (65534/ nobody) Gid: ( 1308/mediawriters)

Etc

16

Inside the container, when as root you issue mkdir /mnt/lxcstorage/Photos/somefolder, what’s the resulting UID/GID on somefolder?

17

root@syncthing: mkdir /mnt/lxcstorage/photos/somefolder/testfolder && ls -l

drwxr-xr-x 2 root root 4096 mmm dd hh:mm testfolder

and in the host, just for completeness:

root@pvehost cd /mnt/hoststorage/photos/somefolder && ls -l

drwxr-xr-x 2 100000 100000 4096 mmm dd hh:mm testfolder

Jt seems like removing root from the root group and just leaving it as mediawriters might make the difference? I’m just not sure if that would mess things up on the LXC? Or is it possible to change the default group/permissions that a user writes with somehow?

root@syncthing: id root

uid=0(root) gid=0(root) groups=0(root),1308(mediawriters)

(Thank you for all your help so far, by the way)

18

Thanks. Right after I logged off I realized I’d forgotten to ask about the host side, but was too lazy to log in again (with 2FA). :grinning:

So that’s what I expected the results to be. root inside the container requires that any file/directory it writes/creates be UID:GID 100000:100000.

In your original post there was a snippet from Syncthing’s log:

Is that typo?

Via the bind mount, /mnt/hoststorage/photos and /mnt/lxcstorage/Photos aren’t the same directory.

Although the mkdir by root succeeded inside the container, it’s not the same path as what Syncthing was trying to use.

Definitely not a good idea to remove the root user from the root group. I’ve never tried it, but I know it’ll cause all kinds of issues.

root inside the container belongs to the root group. Although root is also a member of the mediawriters, it’s a supplemental group – anything that root creates will still default to its primary group, i.e., UID:GID 0:0 inside the container and 100000:100000 outside of the container.

The only way to change the default group that’s used when creating files/directories is to modify a user’s primary group membership. Alternatively, you could use the newgrp command to temporarily switch groups and launch Syncthing inside the subshell.

You’re welcome. :nerd_face:

19

I got excited there, I thought we’d found it. Annoyingly, a much more boring answer. The photos/Photos was a typo on my part. The other mix of paths was that was that I have several directories in syncthing, all of which are failing with the same error. I’d just picked the first couple from the logs for brevity, assuming they would be in the same order each time - they weren’t and I hadn’t realised I was jumping between them when posting logs after reboots. I admit, my file organisation could do with some work :rofl:

I’ll pick one of the error log directories to go through step by step here in sequence just to make absolutely sure every step has been tried on the exact same file, exact same location, etc - and keep all the information here in the same comment. Hopefully we’ll find something…

Syncthing Error:

Loading ignores: lstat /mnt/lxcstorage/Photos/User/UserPictures/.stignore: permission denied

Error on folder “User Pictures” (xxxx-xxxx): stat /mnt/lxcstorage/Photos/User/UserPictures: permission denied

Folder / File to test:

/mnt/lxcstorage/Photos/User/UserPictures
/mnt/lxcstorage/Photos/User/UserPictures/.stignore

From the LXC console:

root@syncthing: touch /mnt/lxcstorage/Photos/User/UserPictures
root@syncthing:

root@syncthing: touch /mnt/lxcstorage/Photos/User/UserPictures/.stignore
root@syncthing:

root@syncthing: stat /mnt/lxcstorage/Photos/User/UserPictures

Size: 12288 Blocks: 24 IO Block: 4096 directory
Device: 8,18 Inode: 85622785 Links: 20
Access: (0774/drwxrwxr–) Uid: (65534/ nobody) Gid: ( 1308/mediawriters)

root@syncthing: stat /mnt/lxcstorage/Photos/User/UserPictures/.stignore

Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 8,18 Inode: 85600375 Links: 1
Access: (0774/-rwxrwxr–) Uid: (65534/ nobody) Gid: ( 1308/mediawriters)

Check the syncthing root user is in the right groups:

root@syncthing: id root
uid=0(root) gid=0(root) groups=0(root),1308(mediawriters)

root@syncthing: getent mediawriters
mediawriters:x:1308:root

Create test folder and test file in “permission denied” folder:

root@syncthing: # mkdir /mnt/lxcstorage/Photos/User/UserPictures/TestFolder && touch /mnt/lxcstorage/Photos/User/UserPictures/TestFolder/testfile.txt
root@syncthing:

Check permissions on both from container

root@syncthing: stat /mnt/lxcstorage/Photos/User/UserPictures/TestFolder

Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 8,18 Inode: 89391106 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)

root@syncthing: stat /mnt/lxcstorage/Photos/User/UserPictures/TestFolder/testfile.txt

Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 8,18 Inode: 89391107 Links: 1
Access: (0644/-rw-r–r–) Uid: ( 0/ root) Gid: ( 0/ root)

Inside the host

Check the owner of the ‘permission denied’ file/folder:

root@pvehost: stat /mnt/hoststorage/Photos/User/UserPictures

Size: 12288 Blocks: 24 IO Block: 4096 directory
Device: 8,18 Inode: 85622785 Links: 21
Access: (0774/drwxrwxr–) Uid: ( 1308/mediawriter) Gid: ( 1308/mediawriters)

root@pvehost: stat /mnt/hoststorage/Photos/User/UserPictures/.stignore

Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 8,18 Inode: 85600375 Links: 1
Access: (0774/-rwxrwxr–) Uid: ( 1308/mediawriter) Gid: ( 1308/mediawriters)

Check ownership of the test folder/files created earlier:

root@pvehost: stat /mnt/hoststorage/Photos/User/UserPictures/TestFolder

Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 8,18 Inode: 89391106 Links: 2
Access: (0755/drwxr-xr-x) Uid: (100000/ UNKNOWN) Gid: (100000/ UNKNOWN)

root@pvehost: stat /mnt/hoststorage/Photos/User/UserPictures/TestFolder/testfile.txt

Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 8,18 Inode: 89391107 Links: 1
Access: (0644/-rw-r–r–) Uid: (100000/ UNKNOWN) Gid: (100000/ UNKNOWN)

Check the process owner from inside the host:

root@pvehost: ps -efw | grep syncthing

100000 848821 826707 0 10:56 ? 00:00:00 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
100000 848828 848821 2 10:56 ? 00:00:03 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
root 849356 849256 0 10:57 pts/2 00:00:00 grep syncthing

Check the host user/group membership:

root@pvehost: getent group mediawriters

mediawriters:x:1308:root

root@pvehost: id root

uid=0(root) gid=0(root) groups=0(root),1308(mediawriters)

root@pvehost: id mediawriter

uid=1308(mediawriter) gid=1308(mediawriters) groups=1308(mediawriters)

I think this is everything we tried so far - all collated and listed in one place on the same commenet.

Things I’ve noticed:

root@pvehost: getent group mediawriters

mediawriters:x:1308:root

for some reason, ‘getent group mediawriters’ doesn’t list ‘mediawriter’ as a group member on the host, even though ‘id mediawriter’ lists it as part of the mediawriters group.

root@pvehost: id mediawriter

uid=1308(mediawriter) gid=1308(mediawriters) groups=1308(mediawriters)

I got very excited about this, removed root from the group, added mediawriter and root back to the group again, then relisted the membership. It now shows as a member:

root@pvehost: # getent group mediawriters

mediawriters:x:1308:mediawriter,root

Unfortunately, while I have no idea what could have caused that to happen in the first place, it didn’t solve the problem. Nothing else seems to have changed.

The only other thing I’ve noticed is that syncthing is throwing the error

lstat /mnt/lxcstorage/Photos/User/UserPictures/.stignore permission denied (i.e. lstat on the file, not stat)

Running specifically that command with lstat on the file in the lxc container gives me

root@syncthing: lstat /mnt/lxcstorage/Photos/User/UserPictures/.stignore

-bash: lstat: command not found

I’m not sure whether this is relevant, since it doesn’t seem to solve the other permissions issue (stat, rather than lstat, is shown in the permissions denied error for the directory /mnt/lxcstorage/Photos/User/UserPictures, and stat works fine in the console) but it’s the only other oddity i’ve noticed. Everything else looks absolutely fine and I’m a bit baffled.

I’m going to try adding a user 1308 mediawriter on the container, mapping it through to 1308 mediawriter on the host, and adding it to 1308 mediawriters group on the LXC. Wondering if that will affect the result of the file ownership being “nobody:mediawriters” inside the container and if that will help. Will post another reply with the results soon.

20

I’m going to try adding a user 1308 mediawriter on the container, mapping it through to 1308 mediawriter on the host, and adding it to 1308 mediawriters group on the LXC. Wondering if that will affect the result of the file ownership being “nobody:mediawriters” inside the container and if that will help. Will post another reply with the results soon.

So, mixed news, but mostly bad. Have created on the LXC a new user uid=1308 mediawriter, mapped it to uid=1308 mediawriter on the host, and made its primary group gid=1308 mediawriters.

root@syncthing: getent group mediawriters

mediawriters:x:1308:root,mediawriter

root@syncthing: id mediawriter

uid=1308(mediawriter),1308(mediawriters)

As hoped, this did make the owner of the ‘permission denied’ folders show up as 1308 mediawriter instead of “65534/ nobody”. Running stat again now gives:

root@syncthing: stat /mnt/lxcstorage/Photos/User/UserPictures

Size: 12288 Blocks: 24 IO Block: 4096 directory
Device: 8,18 Inode: 85622785 Links: 20
Access: (0774/drwxrwxr–) Uid: ( 1308/mediawriter) Gid: ( 1308/mediawriters)

root@syncthing: stat /mnt/lxcstorage/Photos/User/UserPictures/.stignore

Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 8,18 Inode: 85600375 Links: 1
Access: (0774/-rwxrwxr–) Uid: ( 1308/mediawriter) Gid: ( 1308/mediawriters)

Unfortunately, this didn’t actually solve the underlying problem: Syncthing still won’t sync anything and still has the same error. Maybe if I try to map root on the LXC to mediawriter on the Host instead…

21

Maybe if I try to map root on the LXC to mediawriter on the Host instead…

Okay it didn’t like that at all. Container starts, and runs but won’t log in, and nothing runs inside it. Syncthing GUI inaccessible. In hindsight should have seen that coming.

[FAILED] Failed to start systemd-logind.service - User Login Management
(from the LXC container logs)

I’ve put it back to just passing through uid=1308 and now i’m totally stumped and out of ideas.

Edit: Okay, seriously weird new development. I had an idea and thought, screw it, I’ll just make the syncthing root the owner of all the files and stop trying to pass through permissions at all. See if that at least solves things in the short term.

root@syncthing: chown -R root:mediawriters /mnt/lxcstorage/Photos
root@syncthing:

This successfully changed the owner, confirmed by

root@syncthing: stat /mnt/lxcstorage/Photos/User/UserPictures

Size: 12288 Blocks: 24 IO Block: 4096 directory
Device: 8,18 Inode: 85622785 Links: 20
Access: (0774/drwxrwxr–) Uid: ( 0/ root) Gid: ( 1308/mediawriters)

root@syncthing: stat /mnt/lxcstorage/Photos/User/UserPictures/.stignore

Size: 0 Blocks: 0 IO Block: 4096 regular empty file Device: 8,18 Inode: 85600375 Links: 1 Access: (0774/-rwxrwxr–) Uid: ( 0/root) Gid: ( 1308/mediawriters)

But syncthing STILL won’t run!

So the the root user of the syncthing LXC now owns these files. I’ve cut away all attempts to pass through permissions. And it’s still throwing the same permission denied errors at me. That’s seriously, seriously weird.

(and yes, syncthing is definitely running as root. Chown root:root instead of root:mediawriters didn’t help anything either)

root@syncthing: ps aux | grep syncthing

root 326 0.0 0.8 1249208 17024 ? Ssl hh:mm 0:00 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
root 333 0.0 2.5 1251384 53236 ? SNl hh:mm 0:03 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0 root 354 0.0 0.0 3324 1536 pts/1 S+ hh:mm 0:00 grep syncthing

22

Oh my god. Got it.

Syncthing was running with systemd. Systemd doesn’t run with supplementary groups (and their permissions) unless specifically specified in the config file. Adding group=mediawriter to the syncthing systemd config file makes it work.

I’m not sure if I can mark this as closed/resolved. But, thankfully, it now is.

3 Likes
23

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.