Swapped to a TrueNAS from a WD MyCloud - Syncthing now reporting a vague error (tmp file cannot be open, operation not permitted).
I’ve got syncthing running in a container on a VM outside of TrueNAS as I didn’t want to couple my NAS with the applications itself.
Couple of things:
Sharing to syncthing via Truenas / NFS share
Confirmed via console in the syncthing container that the share is reachable, mountable and I can read/write (touch test / rm test) with no issues. I did this for root and the user syncthing is running as.
I re-created the folder to share and I ticked ‘Ignore Permissions’ as I read that was an issue as well.
The error:
syncing: finishing: opening temp file: open /media/Downloads/completed/tv-sonarr/filename/.syncthing.filename.srr.tmp: operation not permitted
It’s way too vague for me to actually diagnose what’s going on. I do not think it’s a permissions issue, but potentially a filename / character issue?
Not sure if other TrueNAS & Syncthing users have run into the same thing or not.
If it’s TrueNAS Scale, then internally it’s a GNU/Linux system.
Note that a “container” vs. a “container in/on a VM” are two very different things…
Container:
[Linux [Docker/OCI container] ]
Container in/on a VM:
[Linux [Linux VM [Docker/OCI container] ] ]
Both are valid setups and the choice depends on security, kernel version, and other requirements.
One caveat is that Syncthing’s filesystem watcher feature won’t work across a NFS mount, so it’ll have to rely on the rescan interval to detect changes.
In general, network filesystems such as NFS, SMB/CIFS and so on don’t pass filesystem notifications thru between client and server (there’s one exception that I’m aware of, but there’s a chain of requirements for it to work).
If the devices are the same OS, then syncing permissions will work under the right conditions.
It’s a permissions issue…
Simply repeat your step #2, but with /media/Downloads/completed/tv-sonarr/filename/. If the permissions are indeed correct, then the Syncthing user will be able to read/write files in that path.
Linux filesystems fully support unicode, the only two characters not allowed are null and /, plus there are no reserved words, so even some of the strangest filenames will work fine.
(I don’t see anything in /media/Downloads/completed/tv-sonarr/filename/.syncthing.filename.srr.tmp that’s not acceptable to Linux – not even if the colon was included after “tmp” – which isn’t allowed in Windows.)
It’s been a pretty common theme among those who are using TrueNAS, UnRaid, Synology and other NAS appliances where Syncthing is running inside a container. The issue is due to being unfamiliar with managing all of the layers of permissions rather than anything to do with Syncthing in particular.
Thanks so much for the lengthy response! So a couple of things (that are mostly irrelevant, but please let me know):
I went from TrueNAS Core (when I bought it out of the box) to TrueNAS scale, hoping that Jailing syncthing would solve my issue. Given how noob I am at *nix permissioning, this proved to be a bigger headache than it was worth [I couldn’t even get the Jailed app to even see my mount even though they are on the same dataset], so I fell back to my VM + Container outside of the NAS instance of Syncthing.
The VM runs Ubuntu 22.04, the container seems to run Alpine Linux
Re:
I did this successfully, I should have noted that in the post itself, this was also one of my original thoughts was that it was perms issues all the way down, but having been able to delete any file under that folder and create new files, I wasn’t convinced.
I ran the touch + delete commands as ‘abc’ the user I believe runs syncthing [edit: confirmed with a simple ps aux on the container itself], there’s also a user ‘sync’, which I also ran the above commands on with 0 issue.
From my experience, the things that a user thinks are irrelevant are most often the most important details.
Note that TrueNAS Core is based on FreeBSD while TrueNAS Scale is GNU/Linux under the hood.
It’s an important distinction because FreeBSD has its own container system based on its “jail” subsystem which Linux’s “LXC” has similar goals as, but the designs and setup are quite different.
So when someone says “Jailing”, it generally infers that they’re using FreeBSD.
Note that mounting NFS shares directly from inside of a Docker or LXC container isn’t possible because NFS requires access to /dev which the virtualized environment doesn’t completely replicate.
I’m guessing the reason you’ve got Ubuntu in a VM is because your host system is Windows? Possibly using WSL2?
(Docker/OCI relies on Linux’s “cgroups” which doesn’t exist on Windows and macOS, so a GNU/Linux system is required for hosting containers.)
Given the additional details, it’s not clear how you’re accessing the NFS share from the Syncthing container.
So if I understand correctly, this is the topology:
[VM host [Ubuntu VM [Docker-Syncthing]]] <-> [NFS via TrueNAS Scale]
“on the container”… as in the user running the container? Or was it the user inside the container?
A frequent assumption I’ve seen users new to containers make is thinking that creating a user with the same UID:GID pair inside and outside of a container solves the permissions requirements – it doesn’t.
By necessity, a user inside a container is in a completely different namespace, otherwise it’d result in one of the security issues that users using containers are trying to avoid in the first place – i.e. What’s to stop a non-root user from launching a VM which they have the root login for? They could effectively pretend to be anyone on the host system.
So if user abc inside a container running Syncthing has the UID:GID 1234:1234, it gets mapped to a subordinate UID and GID pair. So the user inside the container ends up with an effective UID:GID of 1001234:1001234 on the host (the exact offset may vary from system to system).
For your particular setup, since you cannot mount the NFS share directly from within the container, you’d have to first mount it on the Ubuntu VM host side and then bind mount it into the container. It means taming NFS permissions, mount point permissions on the Ubuntu side, and bind mount permissions inside the container for Alpine Linux, all while tackling the namespace security.
Personally, I’d scrap the Docker container on the Ubuntu VM.
Syncthing is a single binary executable, so while there are valid use-cases, I think for most users it’s over complicating things wrapping Syncthing in a container. If their “Docker-fu” is strong, then by all means go for it, but otherwise complexity is often the enemy of success.
If the ultimate goal is to run Syncthing on TrueNAS Scale, last time I checked, the officially supported method is a Docker container.
That’s fine, but don’t access the storage volumes via NFS. You lose the benefit of filesystem notifications in Syncthing, and there’s unnecessary system overhead. Instead, use Docker’s volume mapping to bind mount the external storage into the container. It’s more efficient and requires one less layer of permissions to deal with.
Thanks - I think I’m likely complicated this even more because I’m using Portainer to manage my container.
You’ve got it right that:
[VM host [Ubuntu VM [Portainer/Docker-Syncthing]]] <-> [NFS via TrueNAS Scale]
I was using Portainer’s volume mapping - which seemed to work well for other apps like Sonarr/Radarr, it was only Syncthing that’s given me issue.
I’m now trying to see if going via CIFS is better, which I seem to get more progress as I’m hitting into permission denied issues (which I think I can work around)
I’ll look into Docker’s volume mapping as well - but I suspect that’s what Portainer is doing under the hood.
So…I’m an idiot and I’ve figured out my problem, but did a lot of weird things as well.
I decided to swap to CIFS / SMB instead of NFS.
I decided to mount on the host then bind the folder in Portainer/Docker rather than trying to mount the remote folder
Both things made it work flawlessly.
It likely was a permissions issue and like you said, it may have been Docker doing some weird stuff on mount, I’ll never know (and to be honest, I don’t really care).
