syncthing service crashes in iocage jail on freeBSD (TrueNAS CORE)

I can’t start syncthing service after following this steps syncthing - FreeBSD Wiki

After I call service syncthing start there is no errors and no messages, I can find.

What else I can try?

I think you’ll need to find a way to run it so that you get some sort of output from something to indicate what the problem is.

Thanks for replying! What I found so far is: When I ran cli sync thing I see it is running appears normally. But as a service crash’s, with out any messages anywhere I can find

Sounds more like it’s not even starting it than that it’s crashing.

What do service -e and service syncthing status report?

root@syncthing:~ # service -e
/etc/rc.d/cleanvar
/etc/rc.d/ip6addrctl
/etc/rc.d/netif
/etc/rc.d/rtsold
/etc/rc.d/newsyslog
/etc/rc.d/virecover
/etc/rc.d/os-release
/etc/rc.d/cleartmp
/etc/rc.d/motd
/etc/rc.d/syslogd
/usr/local/etc/rc.d/syncthing
/etc/rc.d/cron
root@syncthing:~ #  service syncthing status
syncthing is not running.
root@syncthing:~ #

What’s in the RC file above?

root@syncthing:~ # cat /usr/local/etc/rc.d/syncthing
#!/bin/sh

# PROVIDE: syncthing
# REQUIRE: DAEMON
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
#
# syncthing_enable (bool):      Set to NO by default.
#                               Set it to YES to enable syncthing.
# syncthing_home (path):        Directory where syncthing configuration
#                               data is stored.
#                               Default: /usr/local/etc/syncthing
# syncthing_log_file (path):    Syncthing log file
#                               Default: /var/log/syncthing.log
# syncthing_user (user):        Set user to run syncthing.
#                               Default is "syncthing".
# syncthing_group (group):      Set group to run syncthing.
#                               Default is "syncthing".

. /etc/rc.subr

name=syncthing
rcvar=syncthing_enable

load_rc_config $name

: ${syncthing_enable:="NO"}
: ${syncthing_home=/usr/local/etc/syncthing}
: ${syncthing_log_file=/var/log/syncthing.log}
: ${syncthing_user:="syncthing"}
: ${syncthing_group=${syncthing_group:-$syncthing_user}}

pidfile=/var/run/syncthing.pid
procname="/usr/local/bin/syncthing"
command="/usr/sbin/daemon"
command_args="-cf -p ${pidfile} ${procname} ${syncthing_home:+-home=${syncthing_home}} ${syncthing_log_file:+-logfile=${syncthing_log_file}} -no-browser ${syncthing_args}"

start_precmd=syncthing_startprecmd

syncthing_startprecmd()
{
        if [ ! -e ${pidfile} ]; then
                install -o ${syncthing_user} -g ${syncthing_group} /dev/null ${pidfile};
        fi

        if [ ! -d ${syncthing_home} ]; then
                install -d -o ${syncthing_user} -g ${syncthing_group} ${syncthing_home}
        fi

        if [ ! -e ${syncthing_log_file} ]; then
                install -o ${syncthing_user} -g ${syncthing_group} /dev/null ${syncthing_log_file};
        fi
}

run_rc_command "$1"
root@syncthing:~ #

According to its RC file, Syncthing’s console output will be written to /var/log/syncthing.log.

Even if Syncthing shuts down right after starting, there will still be something in that log file. But if the file doesn’t exist, then there’s an issue with the Syncthing service. Either the service hasn’t been enabled, or the configuration is incomplete.

Here is what I got in the /var/log

And I guess that file was created with some issues since the system believes it is executable. And size is 0 so there is no data in it.

root@syncthing:~ # ls -lF /var/log/
total 53
-rw-------  1 root       wheel        194 Aug 24 15:19 auth.log
-rw-------  1 root       wheel      45384 Aug 24 17:00 cron
-rw-r--r--  1 root       wheel         66 Aug 23 13:22 daemon.log
-rw-------  1 root       wheel         66 Aug 23 13:22 debug.log
-rw-r--r--  1 root       wheel         66 Aug 23 13:22 devd.log
-rw-------  1 root       wheel          0 Aug 24 03:01 dmesg.today
-rw-r--r--  1 root       wheel         66 Aug 23 13:22 lpd-errs
-rw-r-----  1 root       wheel       2295 Aug 24 04:15 maillog
-rw-r-----  1 root       wheel        137 Aug 24 00:00 maillog.0.bz2
-rw-r--r--  1 root       wheel        671 Aug 24 15:20 messages
-rw-------  1 root       wheel         62 Aug 24 03:01 mount.today
-rw-r-----  1 root       network       66 Aug 23 13:22 ppp.log
-rw-------  1 root       wheel         66 Aug 23 13:22 security
-rw-------  1 root       wheel       2956 Aug 24 03:01 setuid.today
-rwxr-xr-x  1 syncthing  syncthing      0 Aug 23 13:24 syncthing.log*
-rw-------  1 root       wheel        215 Aug 23 13:27 userlog
-rw-r--r--  1 root       wheel        197 Aug 24 15:19 utx.lastlogin
-rw-r--r--  1 root       wheel        156 Aug 24 15:19 utx.log

Here is what I found in messages log

root@syncthing:~ # cat /var/log/messages
Aug 23 13:22:03 syncthing newsyslog[63607]: logfile first created
Aug 23 13:22:03 syncthing syslogd: kernel boot file is /boot/kernel/kernel
Aug 23 13:22:49 syncthing pkg-static[63801]: pkg-1.21.3 installed
Aug 23 13:23:03 syncthing pkg[63824]: syncthing-1.27.9_1 installed
Aug 24 15:20:22 syncthing root[85676]: /usr/sbin/service: WARNING: $syncthingrelaysrv_enable is not set properly - see rc.conf(5).
Aug 24 15:20:22 syncthing root[85684]: /usr/sbin/service: WARNING: $syncthingrelaypoolsrv_enable is not set properly - see rc.conf(5).
Aug 24 15:20:22 syncthing root[85692]: /usr/sbin/service: WARNING: $syncthingdiscosrv_enable is not set properly - see rc.conf(5).
Aug 24 17:10:54 syncthing nologin[87107]: Attempted login by root on /dev/pts/16

Looks like you’ve got some issues in /etc/rc.conf or /etc/rc.conf.local

root@syncthing:~ # cat /etc/rc.conf
cron_flags="$cron_flags -J 15"

# Disable Sendmail by default
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Run secure syslog
syslogd_flags="-c -ss"

# Enable IPv6
ipv6_activate_all_interfaces="YES"
rtsold_enable="YES"
ifconfig_epair0b_ipv6="inet6 auto_linklocal accept_rtadv autoconf"

syncthing_enable="YES"

root@syncthing:~ #

The contents of /etc/rc.conf look fine.

Out of curiosity, I dusted off the cobwebs in the FreeBSD room of my brain, downloaded the latest ISO image, and installed it to take a closer look. A couple minutes after the first reboot and I had Syncthing up and running via a RC service.

The contents of /var/log/syncthing.log looked as expected. /usr/local/etc/rc.d/syncthing matched the one in earlier post, and Syncthing’s home directory /usr/local/etc/syncthing/ was the typical collection of files.

The subject line of your post mentions FreeBSD jail, but it doesn’t appear you’re using one based on your followup posts.

I didn’t test Syncthing in a FreeBSD jail, but other than a little extra work in the setup, I wouldn’t expect any runtime issues (I’ve used Syncthing bare-metal, in an OCI container with Docker and Podman, LXC and KVM plenty of times without any issues).

What I’d do is get Syncthing running as a service without a FreeBSD jail first if you haven’t already done that.

2 Likes

Thanks for helping me!

I have a question, why do you think I am not running those commands in jail?

I went to jail tab in TrueNAS web interface, selected the jail I created for syncthing and pressed “shell” for my understanding it lunched the shell inside of the jail.

Also inside of the jail when I call syncthing it starts and works normally and I can open GUI and there is no issues.

There is something wrong with running it as a service and can’t get a clue what.

Although the limited list of services from service -e looked like it might be from within a FreeBSD jail, it could have also been a stripped down system, and there had been no mention of the container actually working, its configuration, etc. :smirk:

I didn’t notice any issues with your /etc/rc.conf and other info, but it’s still very curious that you’re able to start Syncthing manually – I’m assuming via a service syncthing start – without there being any log file. Perhaps you’ve got more than one Syncthing configuration.

My own test results with FreeBSD jail…

It’d been ages since I played around with FreeBSD jail, so it was interesting to see how it’s changed since I’d last seen it (still reminds me of UML).

After setting up a sample jail named “syncthing” → switching into the jail → installing the Syncthing package from the FreeBSD repo → adding a service file in /etc/rc.d/ → issuing service syncthing enable → restarting the jail…

Checking for Syncthing from the host side…

root@freebsd:~ # ps -uax | grep syncthing
syncthing 11570   2.0  0.9 1269396 38332  -  SNJ  00:15     0:01.46 /usr/local/bin/syncthing -home=/usr/local/etc/syncthing -logfile=/var/log/syncthing.log -no-browser
syncthing 11546   0.0  0.1   12828  2532  -  SsJ  00:15     0:00.00 daemon: /usr/local/bin/syncthing[11547] (daemon)
syncthing 11547   0.0  0.7 1264148 28964  -  SJ   00:15     0:00.02 /usr/local/bin/syncthing -home=/usr/local/etc/syncthing -logfile=/var/log/syncthing.log -no-browser
root      11584   0.0  0.1   12808  2340  0  S+   00:15     0:00.00 grep syncthing

Logging into the jail and checking things within…

root@syncthing:/ # service syncthing status
syncthing is running as pid 11547.
root@syncthing:/ # head /var/log/syncthing.log 
[start] 2024/08/27 00:15:13 INFO: syncthing v1.27.9 "Gold Grasshopper" (go1.21.12 freebsd-amd64) ports@freebsd 2024-08-08 16:32:18 UTC [noupgrade]
[start] 2024/08/27 00:15:13 INFO: Generating ECDSA key and certificate for syncthing...
[start] 2024/08/27 00:15:13 INFO: Default folder created and/or linked to new config
[start] 2024/08/27 00:15:13 INFO: Default config saved. Edit /usr/local/etc/syncthing/config.xml to taste (with Syncthing stopped) or use the GUI
[start] 2024/08/27 00:15:13 INFO: Archiving a copy of old config file format at: /usr/local/etc/syncthing/config.xml.v0
[4KUBL] 2024/08/27 00:15:13 INFO: My ID: 4KUBLPI-N6Z63XS-WB6F3AN-WHMI3LC-RS7SZGL-K3OGGFU-JH6F6K2-V7AOOAB
[4KUBL] 2024/08/27 00:15:14 INFO: Single thread SHA256 performance is 186 MB/s using crypto/sha256 (186 MB/s using minio/sha256-simd).
[4KUBL] 2024/08/27 00:15:15 INFO: Hashing performance is 165.55 MB/s
[4KUBL] 2024/08/27 00:15:15 INFO: Running database migration 1...
[4KUBL] 2024/08/27 00:15:15 INFO: Running database migration 2...

Finally, existing the jail and doing a quick probe to see if Syncthing’s web GUI is available…

root@freebsd:~ # curl -I http://127.0.0.1:8384/
HTTP/1.1 200 OK
Cache-Control: no-cache, must-revalidate
Content-Length: 74925
Content-Type: text/html; charset=utf-8
Etag: "66cd60e3"
Last-Modified: Tue, 27 Aug 2024 05:15:15 GMT
Set-Cookie: CSRF-Token-4KUBLPI=PDJccHGvosGXpeEU72MPumwEYRKAbQymZCYsNqeRSE5gAjc29MEMaFSyPKuUpwPC
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Syncthing-Id: 4KUBLPI-N6Z63XS-WB6F3AN-WHMI3LC-RS7SZGL-K3OGGFU-JH6F6K2-V7AOOAB
X-Syncthing-Version: v1.27.9
X-Xss-Protection: 1; mode=block
Date: Tue, 27 Aug 2024 05:16:33 GMT

Since I started with the FreeBSD base for the jail, comparing the two binaries…

root@freebsd:~ # md5sum /usr/local/bin/syncthing /usr/local/jails/containers/syncthing/usr/local/bin/syncthing 
a7cc12e7fc4aeee6ba58f63b7695f574  /usr/local/bin/syncthing
a7cc12e7fc4aeee6ba58f63b7695f574  /usr/local/jails/containers/syncthing/usr/local/bin/syncthing

So at least with a FreeBSD jail on a stock FreeBSD system, no issues whatsoever with autostarting Syncthing via a RC service.

Ok, I did update my TrueNAS to latest release 13.3 and recreated the jail and successfully install Syncthing in it. There was some issue with jail but I have no idea what it was.

Thanks everyone! I am back to syncing my files!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.