Relay server stopped connecting

Support
1

Hello,

the public relay server on my Raspberry Pi 2B (running Ubuntu Server 20.04.1 LTS) stopped working a few days ago. I can still access the status page but it shows zero sessions and connections.

$ sudo systemctl status strelaysrv.service

● strelaysrv.service - Syncthing Relay Server
     Loaded: loaded (/lib/systemd/system/strelaysrv.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-10-29 23:24:35 UTC; 22s ago
       Docs: man:strelaysrv(1)
   Main PID: 12619 (strelaysrv)
      Tasks: 13 (limit: 1827)
     CGroup: /system.slice/strelaysrv.service
             └─12619 /usr/bin/strelaysrv -nat=false -global-rate 3145728 -provided-by 🌠 LorTorDeRMur--Bochum, GER

Okt 29 23:24:35 raspi2b systemd[1]: Started Syncthing Relay Server.
Okt 29 23:24:36 raspi2b strelaysrv[12619]: 2020/10/29 23:24:36 main.go:139: strelaysrv v1.8.0 "Fermium Flea" (go1.13.8 linux-arm) deb@build.syncthing.net 2020-08-07 06:09:12 UTC [noupgrade]
Okt 29 23:24:36 raspi2b strelaysrv[12619]: 2020/10/29 23:24:36 main.go:145: Connection limit 419430
Okt 29 23:24:36 raspi2b strelaysrv[12619]: 2020/10/29 23:24:36 main.go:234: URI: relay://0.0.0.0:22067/?id=I7STPVT-3SGNAFM-SV7GKQM-BPISG5F-WWSBFOQ-UJNT7FV-YQTQ7AT-RFKVCQ4&pingInterval=1m0s&networkTimeout=2m0s&sessionLimitBps=0&g>
Okt 29 23:24:36 raspi2b strelaysrv[12619]: 2020/10/29 23:24:36 main.go:237: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Okt 29 23:24:36 raspi2b strelaysrv[12619]: 2020/10/29 23:24:36 main.go:238: !!  Joining default relay pools, this relay will be available for public use. !!
Okt 29 23:24:36 raspi2b strelaysrv[12619]: 2020/10/29 23:24:36 main.go:239: !!      Use the -pools="" command line option to make the relay private.      !!
Okt 29 23:24:36 raspi2b strelaysrv[12619]: 2020/10/29 23:24:36 main.go:240: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Okt 29 23:24:40 raspi2b strelaysrv[12619]: 2020/10/29 23:24:40 pool.go:70: Failed to join https://relays.syncthing.net/endpoint: request or check error
Okt 29 23:24:40 raspi2b strelaysrv[12619]: 2020/10/29 23:24:40 pool.go:71: Response data: getting invitation: dial tcp 94.222.185.165:22067: i/o timeouta

Errors indicating that the server fails to join the pool appear in the log once every minute.

$ sudo systemctl status strelaysrv.service

● strelaysrv.service - Syncthing Relay Server
     Loaded: loaded (/lib/systemd/system/strelaysrv.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2020-10-29 23:24:35 UTC; 8min ago
       Docs: man:strelaysrv(1)
   Main PID: 12619 (strelaysrv)
      Tasks: 13 (limit: 1827)
     CGroup: /system.slice/strelaysrv.service
             └─12619 /usr/bin/strelaysrv -nat=false -global-rate 3145728 -provided-by 🌠 LorTorDeRMur--Bochum, GER

Okt 29 23:28:48 raspi2b strelaysrv[12619]: 2020/10/29 23:28:48 pool.go:70: Failed to join https://relays.syncthing.net/endpoint: request or check error
Okt 29 23:28:48 raspi2b strelaysrv[12619]: 2020/10/29 23:28:48 pool.go:71: Response data: getting invitation: dial tcp 94.222.185.165:22067: i/o timeout
Okt 29 23:29:50 raspi2b strelaysrv[12619]: 2020/10/29 23:29:50 pool.go:70: Failed to join https://relays.syncthing.net/endpoint: request or check error
Okt 29 23:29:50 raspi2b strelaysrv[12619]: 2020/10/29 23:29:50 pool.go:71: Response data: getting invitation: dial tcp 94.222.185.165:22067: i/o timeout
Okt 29 23:30:52 raspi2b strelaysrv[12619]: 2020/10/29 23:30:52 pool.go:70: Failed to join https://relays.syncthing.net/endpoint: request or check error
Okt 29 23:30:52 raspi2b strelaysrv[12619]: 2020/10/29 23:30:52 pool.go:71: Response data: getting invitation: dial tcp 94.222.185.165:22067: i/o timeout
Okt 29 23:31:54 raspi2b strelaysrv[12619]: 2020/10/29 23:31:54 pool.go:70: Failed to join https://relays.syncthing.net/endpoint: request or check error
Okt 29 23:31:54 raspi2b strelaysrv[12619]: 2020/10/29 23:31:54 pool.go:71: Response data: getting invitation: dial tcp 94.222.185.165:22067: i/o timeout
Okt 29 23:32:56 raspi2b strelaysrv[12619]: 2020/10/29 23:32:56 pool.go:70: Failed to join https://relays.syncthing.net/endpoint: request or check error
Okt 29 23:32:56 raspi2b strelaysrv[12619]: 2020/10/29 23:32:56 pool.go:71: Response data: getting invitation: dial tcp 94.222.185.165:22067: i/o timeout

I updated several packages and kernels recently (not sure which ones) and I am using the official Syncthing repository. Any ideas?

2

Seems like firewall getting in the way?

Is the relay behind a NAT? Perhaps it times out trying to acquire upnp mappings.

3

I am really clueless here… my hardware setup is such that the RasPi is connected to a modem router via ethernet cable. A UniFi WLAN AP is also connected to the router via ethernet. TCP ports 22067-22070 are forwarded in the router and I also tried setting NAT=true in /etc/default/syncthing-relaysrv.

Here is the file:

# Default settings for syncthing-relaysrv (strelaysrv).
NAT=false

## Add Options here:
RELAYSRV_OPTS=-global-rate 3145728 -provided-by "🌠 LorTorDeRMur--Bochum, GER"

I had a look at /var/log/dpkg.log and the only possibly relevant packages that were installed/updated are systemd and unifi (the AP’s management software).

4

I’m not sure who packaged the relay server for your OS, I don’t think we package it.

I am not sure what the NAT flag does, but from the symptoms imply that the port forwarding doesn’t work.

I’d check the if you can reach those ports from another machine, my gut says that either the port mappings are gone, for the RPI restarted or something and is now firewalling off those ports.

5

According to aptitude the syncthing-relaysrv package is from apt.syncthing.net, thus it appears to be maintained by the Syncthing team.

If I try to access the RasPi externally on port 22070, it just hangs indefinitely.

I remember that the relay stopped connecting after a couple of server reboots in a row. Is there something I can try or do via ufw/iptables or similar?

6

Sorry, my bad, we do. Went looking for a systemd file and it was not in the usual place.

The fact it hangs implies it’s the firewall.

We have some docs around firewalls in generic syncthing docs which might be transferable here, but given there are quite a few different implementations of firewalls, guides on how to configure them are out of scope for syncthing. You know which ports need to be accessible, I guess you need to find a guide internet on how to configure the one you are using.

We do ship a ufw file as part of the deb package, so perhaps it’s something as simple as ufw allow strelaysrv strelaysrv-metrics, but this is just guessing, as I mostly run Windows and I have no idea what type of firewall you are using.

7

Turns out my RasPi 2 had a hardware failure. The ethernet port gradually stopped working until there was no connectivity whatsoever. On powering down the server the SD card was completely busted too.

After reinstalling on a RasPi 3 the relay is now fully functional using only the router’s port forwarding. The downside is that I installed the 64 bit version of Ubuntu Server, and Syncthing itself, while available through the official repository, won’t start on that. So I’m using an rsync cron script instead for the time being.

8

Huh? Which repository of what are you using, and why won’t it start?

9

I am using the apt.syncthing.net repository. Syncthing does not appear to start on boot and I have to issue the syncthing command manually after which everything works properly. I also cannot find a script in /etc/init.d or a systemd service.

$ sudo systemctl enable syncthing
Failed to enable unit: Unit file syncthing.service does not exist.

Grepping the process list after a reboot:

$ ps aux | grep sync
root         110  0.0  0.0      0     0 ?        S<   17:21   0:00 [vchiq-sync/0]
systemd+    1589  0.2  0.5  90128  4956 ?        Ssl  17:22   0:00 /lib/systemd/systemd-timesyncd
syncthi+    1731 10.7  1.0 113820  9812 ?        Ssl  17:22   0:23 /usr/bin/strelaysrv -nat=false -global-rate 3145728 -provided-by 🌠 LorTorDeRMur--Bochum, GER
ubuntu      2991  1.0  0.0   6292   660 pts/0    S+   17:26   0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox sync

Only the relay server is in the process list after boot.

10

I got it sorted out after reading the FAQ.

The following enabled the Syncthing service, and it persists after a reboot too:

$ systemctl enable syncthing@ubuntu.service
$ systemctl start syncthing@ubuntu.service
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.