Hello. So i had issues with Syncthing right from the start of using it. I’d like to introduce my layout first so everyone here have better image of what my setup looks like.
I do have Ubuntu Server VM on which i run docker with Syncthing server. My backup location is on virtually remote. My other VM is my NAS so i connect to it over SMB with this command:
sudo mount -t cifs -o rw,vers=3.0,credentials=/root/.smbcredentials //192.168.1.72/Backups /mnt/Backups
So i have read and write permissions on my mount with given user credentials. Everything’s fine. I mapped this /mnt/Backups to the Syncthing server so i can backup devices. Now, the error looks like this:
As i attached console and looked into that folder, i didn’t find that file. In fact, i didn’t find any of the temp files that syncthing created. But all the files i see there has rwx permissions on root user so it’s right.
Now for the incident: At the beginning of November, Watchtower docker container screwed up itself. It mindlessly started to creating new containers (ended up with DDOSing my local network as router couldn’t handle such traffic) and filling whole given storage to max. I noticed it couple of days ago after troubleshooting mentioned traffic issue. From that point i’m unable to sync anything to Syncthing server.
It is not caused by having wrong permissions (as it worked well-ish till that incident) nor by special characters in filenames as those did back up normally as well before the incident. Heck, i’m even unable to sync any image shot by my Android 12 phone (or any device really).
What i tried:
Rebooting NAS VM, Ubuntu VM, and Syncthing container itself.
Looking into logs but it did not show anything worthy excluding already mentioned lines.
Any reason for not having the permission?
PS: If you will need any kind of logs, i’m willing to share it (of course).
If the Watchtower container you are using is what I think it is, your Syncthing container might have been upgraded and settings changed when your Watchtower container went haywire, so the first place I would look at is how /mnt/Backups is mapped into your Syncthing container:
Is /mnt/Backups actually set up as a volume for your Syncthing container?
Does the user:group Syncthing is running as inside your container have read/write access to /mnt/Backups.
On a related note…
With your current setup, there really isn’t much benefit to running Syncthing in a container, at least not with regards to security, because /mnt/Backups is mounted as root and writable by your Syncthing container.
It is also not recommended, or supported, using Syncthing to sync to/from a network share. Permissions, network I/O and other issues make such a setup complicated and prone to problems.
Is /mnt/Backups actually set up as a volume for your Syncthing container?
Yes. It’s a binded volume so /mnt/Backups is the same path inside the container.
Does the user:group Syncthing is running as inside your container have read/write access to /mnt/Backups.
Yes. Yesterday I tried to manually create txt file inside the mounted directory and worked flawlessly.
It is also not recommended, or supported, using Syncthing to sync to/from a network share.
But then, in my opinion, it defeats whole purpose of such service. Unless Syncthing is installed locally on TrueNAS for example. Is that an option anyway?
Permissions, network I/O and other issues make such a setup complicated and prone to problems.
I agree with this, although I didn’t find better and easier way to deploy it.
Good, but best to verify from a view inside the container.
That’s great, but that was from the host side, and as root (the mount show earlier command didn’t include any UID/GID options).
What happens when a command shell is attached to the Syncthing container and you try to create the text file on the bind mount from within the container?
Depending on the Syncthing container being used, what user is running Syncthing inside the container and does it have write permissions to the mount point?
There’s actually more than one option, but Syncthing is one of the TrueNAS Core plugins (https://truenas.com/apps), and for TrueNAS Scale, Docker containers.
Syncthing and other similar apps, is peer-to-peer. With a setup such as Ubuntu Server VM → Syncthing → SMB → NAS VM, it’s no longer peer-to-peer because Syncthing is relying on SMB for the final leg of the network trip instead of talking to a Syncthing peer.
Good, but best to verify from a view inside the container.
Yep. I checked it by connecting to interactive session inside the container.
That’s great, but that was from the host side, and as root (the mount show earlier command didn’t include any UID/GID options).
That is actually a good point. But i believe, there’s only a root user which makes all the operations inside the container, right? There might be issue if the operations would be made by different user than root inside the container.
What happens when a command shell is attached to the Syncthing container and you try to create the text file on the bind mount from within the container?
I did exactly that. I described it in my last reply but not deeply, sorry. So yes, i did connect to the container as root user (via Portainer and it’s interactive session). I navigated to that mounted directory and created file flawlessly.
Depending on the Syncthing container being used, what user is running Syncthing inside the container and does it have write permissions to the mount point?
Well, that’s exactly what i discussed at the start of this reply. I’m not sure how to check which user exactly is running Syncthing software itself, inside the container. Any clue how to check it?
That’s awesome. I might switch to that. What is your recommendation?
Syncthing and other similar apps, is peer-to-peer. With a setup such as Ubuntu Server VM → Syncthing → SMB → NAS VM, it’s no longer peer-to-peer because Syncthing is relying on SMB for the final leg of the network trip instead of talking to a Syncthing peer.
Yeah. I understand that. But as i needed it ASAP, i didn’t really mind a perfect setup. Now i understand it’s better as it is more future-proof and issue-proof.
No problem. It only required a few minutes to reread the earlier posts to refresh my memory.
It depends on the particular container. Which Syncthing container are you using? (the URL to the page on Docker Hub would be very helpful).
Some containers have a nearly complete operating system in them, while others have just the bare minimum to run the app inside. For example, the official Syncthing container is a lean 34MB [busybox container], but even it looks bulky compared to the tiny 1.46MB busybox container.
Okay, so go back into the interactive session and find out what user/group Syncthing is running under inside the container. /mnt/Backups on your Ubuntu Server VM must allow the user running Syncthing to read/write to the mount point from within the container.
While you’re attached to the running container with an interactive shell, issue the command ps -ef to see the process list (or alternatively, use the top command).
I don’t use TrueNAS, but from what I can see on the website, it depends on if you’re using TrueNAS Core or TrueNAS Scale. If it’s the former, it’s the plugin. If it’s the latter, then Docker plus a Syncthing container.
And if it’s Docker, stick with the official Syncthing container unless you have a good reason not to because it’s more likely someone else is using the same one on this forum in case you need help.
Personally, I’d keep things simple by running Syncthing directly on the Ubuntu Server VM without layering on Docker and Portainer (Syncthing is in Ubuntu’s official repo and the Syncthing project has its own APT repo).
I think Portainer is great addition for Docker, but unless Syncthing is being added to a swarm of already running containers potentially as part of a pod, it just seems like an overly complicated setup for an app as compact as Syncthing that was also designed to be run by a non-root user.
/mnt/Backups on your Ubuntu Server VM must allow the user running Syncthing to read/write to the mount point from within the container.
Yes, i’m aware of that.
While you’re attached to the running container with an interactive shell, issue the command ps -ef to see the process list (or alternatively, use the top command).
Well, it appears to be same output as it is from top (obviously).
I don’t use TrueNAS, but from what I can see on the website, it depends on if you’re using TrueNAS Core or TrueNAS Scale. If it’s the former, it’s the plugin. If it’s the latter, then Docker plus a Syncthing container.
Agree, i have TrueNAS Core and found it in the “Plugin” tab inside the WebUI.
Personally, I’d keep things simple by running Syncthing directly on the Ubuntu Server VM without layering on Docker and Portainer (Syncthing is in Ubuntu’s official repo and the Syncthing project has its own APT repo).
Well, i could do that but i like to have things organized and also don’t want to fall into dependency hell (again). I think it might be okay to switch from Ubuntu VM to TrueNAS itself with Syncthing so it will have direct access to backup dirs and so on. Also, it is officially supported by iXSystems which is a bonus.
I did some googling but it appears that Syncthing doesn’t have any option to export settings in WebUI. Also, someone said when he reinstalled instance of Syncthing, it did remove all contets of backup folders which would be very unpleasant.
I’m familiar with that particular Syncthing container from LinuxServer.io.
LinuxServer.io has a uniform core across its catalog of containers. I’ve been using their Plex Media Server container for years.
Both LinuxServer.io’s PMS and Syncthing containers run the primary application under a dedicated non-root user named “abc” who belongs to the group “abc” with both the UID and GID set to 911.
So…
/mnt/Backups on your Ubuntu Server VM is owned by root:root (UID=0,GID=0).
Syncthing inside the container is running as abc:abc (UID=911,GID=911).
Therefore, the Syncthing user cannot modify the contents of /mnt/Backups inside or outside of the container. From LinuxServer.io’s Syncthing container page: “Ensure any volume directories on the host are owned by the same user you specify and any permissions issues will vanish like magic.”
I get it, I have similar personal preferences with regards to keeping the servers I manage nice and tidy. However, I also believe that putting everything in separate containers is not always the right solution either if it actually complicates things.
At least on Linux systems, dependency hell comes up most often during software development or when trying to install software binaries from some place other than a repository suitable for the particular Linux distro. Syncthing has no external dependencies (the default compiler setting for Go programs), so running with a container means trading this…
… for this:
(And given that the Ubuntu server is itself a VM, there are even more layers to the stack.)
That’s true, but it’s not necessary to have a settings export feature. Both because the settings aren’t automatically useful on another system, and also because the settings contain API keys and other sensitive info. To back up the settings, simply copy the configuration directory appropriate for the host OS.
Every time I’ve read a user post about losing files after reinstalling Syncthing, the user was running Syncthing in a Docker container and didn’t bother to map any external volumes for Syncthing to write data to. So everything (including the settings) were stored only inside the container. Docker containers are immutable, so nothing persists when the container is shut down.
The problem isn’t Syncthing. It’s too many new users deciding to tackle setting up both Syncthing and Docker at the same time without having ever used either one before – the act of walking on two legs is mastering controlled falling, but first it’s learning to crawl → walk → run, then perhaps, dance.
Be sure to follow the instructions posted on LinuxServer.io’s Syncthing container page about using the “-v /config”, “-v /data1” and “-v /data2” container parameters to map directories from the host for Syncthing to store its configuration files in and the “PUID” and “PGID” environment variables for setting the UID/GID for user “abc”.
It’s very likely that any configuration settings you’ve already done are stored only inside the container since you hadn’t mentioned otherwise. If want/need to keep them, manually back them up first before deleting the container.
Other than that, the whole point of using containers is to isolate apps from the host OS and data. Switching containers for the same app is like changing brands of light bulbs, as long as the socket is the right fit, it just works.
I’ve been using their Plex Media Server container for years.
Me too. Really reliable, low power and i love their GID and UID env variables.
Both LinuxServer.io’s PMS and Syncthing containers run the primary application under a dedicated non-root user named “abc” who belongs to the group “abc” with both the UID and GID set to 911.
Riiiight. Now it makes sense. But i still don’t understand why it worked until that accident happened? It sounds like i got stuck on some weird version of Syncthing container. Weird. Anyway, i will check the permissions. And maybe some way to grand the permissions to “abc” user to that mount? I don’t want to create / share group between root and abc and edit whole mount’s permissions.
(And given that the Ubuntu server is itself a VM, there are even more layers to the stack.)
Understood. It is stupid, i can confirm.
the user was running Syncthing in a Docker container and didn’t bother to map any external volumes for Syncthing to write data to
Welp, that’s clearly their own fault.
Switching containers for the same app is like changing brands of light bulbs, as long as the socket is the right fit, it just works.
Yup. I think i might play a bit with my current setup especially with granting “abc” user needed permissions and if it wouldn’t work for some reason, i would switch to TrueNAS with Syncthing.
It’s only an educated guess, but since LinuxServer.io uses the same core image for all of its containers – and assuming that nothing else changed on the Ubuntu Server VM – then it was likely an inadvertent configuration change while restoring things back to normal.
Most filesystem modules in Linux support overriding the UID and GID on the mount point. I know for certain that the CIFS module does because I’ve used it many times before. So your updated mount command would be:
mount -t cifs -o rw,vers=3.0,uid=911,gid=911,credentials=/root/.smbcredentials //192.168.1.72/Backups /mnt/Backups
Note:
The default mode is read/write, so “rw” isn’t strictly required.
Unless there’s a specific need, it’s better to not include the “vers” option so that Linux and a SMB server will negotiate for the highest supported SMB protocol version for better security, efficiency and features.
No, not stupid, just not always ideal.
Over time as I had to manage with more and more servers I realized that too many moving parts increases the chances of something going wrong – and when it does – finding the cause was that much harder. Complexity is the enemy of security and ease of maintenance.
Over on Duplicacy’s user forum (open-source backup software), I’ve seen a number of posts for help where someone has installed a container with Duplicacy → not mapped any external storage → start backing up files → then question the reliability of Duplicacy when all of their backups disappear after a docker stop.
mount -t cifs -o rw,vers=3.0,uid=911,gid=911,credentials=/root/.smbcredentials //192.168.1.72/Backups /mnt/Backups
The issue here is that i can’t use the command as i’m running it inside docker / portainer. I’d first like to try fix it within the existing setup and if it will be difficult / impossible, switch to TrueNAS.
Over time as I had to manage with more and more servers I realized that too many moving parts increases the chances of something going wrong
I can relate lol.
question the reliability of Duplicacy when all of their backups disappear after a docker stop
People learn from stupidity. At least they should.
EDIT: According to this issue on github, i probably have to go with TrueNAS or on bare Ubuntu VM. The moving process shouldn’t be that hard. Or can be?
What could possibly go wrong and what is the correct way to make the “move” successful? I think i would just install it, configure it again from scratch and pray it is smart enough that it detects some of the files already exists in the remote directories so it wouldn’t backup it again. Is it the right way?
I’m sorry if I misunderstood part of your setup. I was under the impression that the SMB/CIFS mount was on the Ubuntu Server host side rather than inside of the container.
The command would be easier to run on the Ubuntu Server host side and then you simply map /mnt/Backups into the Syncthing container via the “-v” parameter.
No, it wouldn’t be hard with either option.
Correct, Syncthing actually goes even further by not transferring chunks that already exist.
If you could draw a diagram of what you’ve got without Syncthing, what you’ve currently tried to do (and isn’t working), and then what you want (a photo of a hand drawing is fine), it’d be worth a 1,000 words.
So i dove into it more deeply, tried TrueNAS way many times, tried to solve the docker issue and so on. After 3 hours of brainstorming (i count last couple of days), i noticed that i have mounted /mnt/Backups folder on Ubuntu VM as wrong user (root) and i did have it mounted as user that run docker in the past.
At that point i was like "fck me". Umount it and mounted it as the right user. Now it works as it did before the fckup.
Thank you for being helpful and patient in my troubleshoot journey. I really appreciate it.
