Syncthing and systemd

Support
1

I have a problem getting syncthing getting properly running under systemd. Here’s what I did:

  • I have Cubietruck that is running armbian, a debian based system. Lately I upgraded this from jessie to stretch and this seems to include a change to systemd.
  • As the machine shall serve as a server I want to run syncthing as a system service.
  • syncthing and syncthing-inotify are by default run as root and of course I wanted to change that. So I did:
  • Created a user “syncthing”: sudo adduser --quiet --system --group --disabled-password syncthing
  • Restored my old config.xml, cert.pem, etc. from the previous jessie installation
  • Changed /etc/systemd/system/syncthing.service and replaced “User=root” by “User=syncthing”
  • Did the same with /etc/systemd/system/syncthing-inotify.service
  • Disabled the old root start mechanism: systemctl disable syncthing@root.service systemctl disable syncthing-inotify@root.service
  • Enabled syncthing with the new user: systemctl enable syncthing@syncthing.service systemctl enable syncthing-inotify@syncthing.service
  • Started both services: systemctl start syncthing@syncthing.service systemctl start syncthing-inotify@syncthing.service
  • I can - successfully - check now that both services are running: systemctl status syncthing@syncthing.service systemctl status syncthing-inotify@syncthing.service

This works at first glance, however I’m facing several issues:

  • Syncthing is started in /etc/systemd/system/syncthing.service via the command line ExecStart=/usr/bin/syncthing -no-browser -no-restart -logfile=/var/log/syncthing.log -logflags=3 Though there is a “-logfile” command no such file is created, instead it spams into syslog. This happens independently of running it as root or an unpriviledged user. I want a separate log for syncthing, syslog is full of syncthing, hardly anything else to see there.
  • When running unpriviledged the web interface becomes unreachable.
  • When running unprivilegded syncthing-inotify runs into troubles with syncthing, the log becomes full of: “http post forbidden. missing API key”
  • Syncthing starts to reject clients that it should know (and knows when ran as root): “Connection from XXXXXX-XXXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX at [fe80::7e04:eaa8:5756:5cf5%eth0]:48792 (tcp-server) rejected: unknown device” “Failed to exchange Hello messages with XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX at 192.168.178.11:22000-192.168.178.32:39550/tcp-server: EOF”

Thus I was forced for the time being to return to using the root account :-/

I’m suspicious this is mostly a rights and configuration problem, however I don’t know what to check and correct. Could anyone give a complete instruction how to move syncthing from the default root-usage to an unpriviledged user?

THX

Don

2

After writing all of the following, I suddenly thought: “Wait, he updated to jessie? That’s ancient!”. So what version of Syncthing are you running?
The following is still true (and incomplete), but there’s no point going forward with any of it before knowing your Syncthing version.

There is nothing “special” about running Syncthing as an unprivileged user, quite the contrary, it is the “norm” - so I am not sure what the problem is but I noticed some points:

You seem to be using a package specific to Cubietruck/armbian/…, as Syncthing’s systemd service is not run as root by default and is called “syncthing@.service”. It is generic, i.e. the user is decided by what you add after the “@” at "runt

What does the log line “Access the GUI via the following URL: …” say?

You can’t use -no-restart and -logfile together and the current version does abort on startup if you use both.

1 Like
3

After writing all of the following, I suddenly thought: “Wait, he updated to jessie? That’s ancient!”. So what version of Syncthing are you running?

No, you misunderstood! I updated from Jessie to Stretch. I also wrote that: “Lately I upgraded this from jessie to stretch”. So that’s OK, I’m using a current version of the OS as well as syncthing. syncthing is V0.14.44, the Linux kernel is V4.14.18.

There is nothing “special” about running Syncthing as an unprivileged user, quite the contrary, it is the “norm”

Exactly! That how I was running syncthing before the update and I would like to have it that way again.

You seem to be using a package specific to Cubietruck/armbian/

Presumably, yes. I could install syncthing from the repositories that came with the distro, no need to add a new source to apt. So it came with the distro. I have a syncthing.service in /etc/systemd/system with the following content (unchanged by me):

[Unit]
Description=Syncthing - Open Source Continuous File Synchronization
Documentation=man:syncthing(1)
After=network.target

[Service]
ExecStart=/usr/bin/syncthing -no-browser -no-restart -logfile=/var/log/syncthing.log -logflags=3
Restart=on-failure
SuccessExitStatus=3 4
RestartForceExitStatus=3 4
User=root

[Install]
WantedBy=default.target

What does the log line “Access the GUI via the following URL: …” say?

INFO: Access the GUI via the following URL: xxxx://127.0.0.1:8384/ INFO: Access the GUI via the following URL: xxxx://192.168.178.11:8384/ (had to replace the http above with xxx as the forum regards this as a hyperlink)

If you want to know if the web GUI is set to localhost-only the answer is no, in the configuration I placed “0.0.0.0:8384”.

You can’t use -no-restart and -logfile together and the current version does abort on startup if you use both.

This line is written in /etc/systemd/system/syncthing.service that came from the distro. And strangely contrary to what you claim syncthing runs with this line:

systemctl status syncthing@root.service ● syncthing@root.service - Syncthing - Open Source Continuous File Synchronization for root Loaded: loaded (/lib/systemd/system/syncthing@.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-03-01 22:24:08 CET; 23h ago Docs: man:syncthing(1) Main PID: 8856 (syncthing) Tasks: 21 (limit: 4915) CGroup: /system.slice/system-syncthing.slice/syncthing@root.service └─8856 /usr/bin/syncthing -no-browser -no-restart -logflags=0

Mär 02 22:10:13 cubietruck syncthing[8856]: [YTF3N] INFO: Puller (folder “xxx” (xxx), file “xxx”): delete dir: directory contains ignored files (see ignore documentation for (?d) prefix) Mär 02 22:10:13 cubietruck syncthing[8856]: [YTF3N] INFO: Puller (folder “xxx” (xxx), file “xxx”): delete dir: directory contains ignored files (see ignore documentation for (?d) prefix) Mär 02 22:10:13 cubietruck syncthing[8856]: [YTF3N] INFO: Puller (folder “xxx” (xxx), file “xxx”): delete dir: directory contains ignored files (see ignore documentation for (?d) prefix) Mär 02 22:10:22 cubietruck syncthing[8856]: [YTF3N] INFO: Established secure connection to xxx at 192.168.178.11:22000-192.168.178.43:38950/tcp-server (TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305) Mär 02 22:10:22 cubietruck syncthing[8856]: [YTF3N] INFO: Device xxx client is “syncthing v0.14.43” named “xxx” at 192.168.178.11:22000-192.168.178.43:38950/tcp-server Mär 02 22:10:31 cubietruck syncthing[8856]: [YTF3N] INFO: Folder “xxx” (xxx) isn’t making progress. Pausing puller for 1m0s. Mär 02 22:11:00 cubietruck syncthing[8856]: [YTF3N] INFO: Puller (folder “xxx” (xxx), file “xxx”): finisher: pull: peers who had this file went away, or the file has changed while syncing. will Mär 02 22:11:05 cubietruck syncthing[8856]: [YTF3N] INFO: Puller (folder “xxx” (xxx), file “xxx”): finisher: pull: peers who had this file went away, or the file has changed while syncing. will Mär 02 22:11:08 cubietruck syncthing[8856]: [YTF3N] INFO: Puller (folder “xxx” (xxx), file “xxx”): finisher: pull: peers who had this file went away, or the file has changed while syncing. w Mär 02 22:11:50 cubietruck syncthing[8856]: [YTF3N] INFO: Folder “xxx” (xxx) isn’t making progress. Pausing puller for 1h4m0s.

so I am not sure what the problem is

The (main) problem is that in the configuration that came with the distro syncthing runs without a problem (aside from the mentioned log problem). However if I change the configuration to run not as root but instead as a standard user I’m facing several issues as described. Of course I want to resolve these to be able to run syncthing as non-root again.

BR

Don

4

All I’ve ever needed to do on my Debian boxes can be summed up in 4 easy steps:

  1. Install Syncthing, obviously.
  2. Add user who will run Syncthing and have access to files - [username]
  3. Systemctl enable syncthing@[username].service
  4. Systemctl start syncthing@[username].service

(sorry, thought you’d changed some systemd config files, but just re-read and noticed you said you didn’t.)

Hang on, you said you restored the config from your Jessie install- daft question, but you did chown and chmod to the newly created user in Stretch, right? :smiley: can syncthing actually read the config?

It will work under root because that’s stored in roots home folder and root can access all regardless - but you’ll probably find the newly created user doesn’t actually have access to those previous config files.

I read “missing API key” as: I can’t read what’s there, and I don’t have permission to change the file to make a new one.

Sorry, many edits. It’s late here- long day.

5

Just to add:

If you do a:

chown -R syncthing:syncthing /home/syncthing/.config/syncthing

And

chmod -R 750 /home/syncthing/.config/syncthing

Does this solve it?

6

Just enclose them in “`” as well.

Are you sure the the file you quoted is in use (i.e. the same as the one given by systemctl status syncthing?
Because that should abort immediately on startup with -logfile may not be used with -no-restart or STNORESTART (see https://github.com/syncthing/syncthing/blob/master/cmd/syncthing/main.go#L345).

7

I know nothing about running anything as a service, but are you sure you can omit the path to config.xml with -home=<dir> option ? Did you restored your config files into /home/syncthing/.config/syncthing ?

8

Did you restored your config files into /home/syncthing/.config/syncthing?

Yep, did so!

9

Excellent point! htop shows me /usr/bin/syncthing -no-browser -no-restart -logflags=0 and this is not what I was expecting! So I think you’re right, syncthing is started by a different command line. This insight made me search for other syncthing files in /etc and I found /etc/systemd/system/multi-user.target.wants/syncthing@root.service that complies to the commandline I see in htop. This is rather confusing (I’m also a bit new to systemd) and I’m wondering why there is a syncthing.service with an obviously broken configuration. It was installed that way by my distro, it’s not me who broke this up.

So what to do to stop syncthing from using syncthing@root.service and start using a theoretical syncthing@syncthing.service? I mean deleting that file is easy but I want systemd to recognize it correctly. Will a
systemctl disable syncthing@root.service
systemctl disable syncthing-inotify@root.service
systemctl enable syncthing@syncthing.service
systemctl enable syncthing-inotify@syncthing.service
systemctl start syncthing@syncthing.service
systemctl start syncthing-inotify@syncthing.service
do the job to take it away from root and assign it to user syncthing? And should syncthing-inotify also be running as non-root or does it need root rights?

OK, I will try to aasemble all of what the people here wrote and see if that changes anything. Hopefully I will get this up’n’runnin’ now…

THX

Don

10

Find out where you installed syncthing from and open an issue with them. Or just uninstall and use https://apt.syncthing.net/. Whether what you proposed will work or not depends on what files are present and I am not too inclined to debug a messed up 3rd party systemd installation.

11

I can understand that! However it seems I finally got it running - with a strange problem left. What I did:

  • Reset the directory rights as suggested:
    chown -R syncthing:syncthing /home/syncthing/.config/syncthing
    chmod -R 750 /home/syncthing/.config/syncthing
  • Created a dir /var/log/syncthing and set its group and owner to synthing to enable the non-root syncthing to log to that dir.
  • Stopped syncthing’s root services and disabled them:
    systemctl stop syncthing@root.service
    systemctl stop syncthing-inotify@root.service
    systemctl disable syncthing@root.service
    systemctl disable syncthing-inotify@root.service
  • Reenabled them for the user syncthing:
    systemctl enable syncthing@syncthing.service
    systemctl enable syncthing-inotify@syncthing.service
    (this created the respective symlinks)
  • Edited /lib/systemd/system/syncthing@.service and /lib/systemd/system/syncthing-inotify@.service and changed the command lines to
    ExecStart=/usr/bin/syncthing -no-browser -logfile=/var/log/syncthing/syncthing.log -logflags=3
    and
    ExecStart=/usr/bin/syncthing-inotify -logfile=/var/log/syncthing/syncthing-inotify.log -logflags=3 -api="xxxx"
  • Executed systemctl daemon-reload
  • Create a file /etc/logrotate.d/syncthing with the following content:
    /var/log/syncthing/syncthing.log
    /var/log/syncthing/syncthing-inotify.log
    {
    rotate 4
    weekly
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
    invoke-rc.d rsyslog rotate > /dev/null
    endscript
    }
  • And finally I started both services:
    systemctl start syncthing@syncthing.service
    systemctl start syncthing-inotify@syncthing.service

This all works, it’s syncing, the web interface is accessible, everyting’s fine. :smile:

The only quirk I currently have is that both logfiles, syncthing.log and syncthing-inotify.log have been successfully created in /etc/log/syncthing but they stay empty. It seems it is not that syncthing doesn’t try to log something there, apparently something’s constantly deleting it. I judge that from the fact that I can see that both files are constantly being touched as the modify time of both files is always “up to date”, means it’s always set to the current time. Any idea what could cause this?

THX

Don

1 Like
12

Meanwhile I found out what caused that: The Cubietruck is a very small machine and it’s booting from an SD-card. In order to minimize writes to the card the distro comes with a tool called log2ram. It redirects /var/log to a ram disk that is cyclically written to SD via a cron job. The problem here is that the space reserved in RAM for the log is not too big, by default it’s 40M. The bunch of log entries syncthing generates completely flooded this fs. Plus it blocked logrotate from working:

error: Compressing program wrote following message to stderr when compressing log /var/log/syslog.1:
gzip: stdout: No space left on device
error: failed to compress log /var/log/syslog.1

Without manual interaction this is blocking forever. I meanwhile solved the problem and logging is operational again, so this is solved.

2 Likes
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.