Hello everybody. I’ll try to be brief, but I have a really weird and specific issue with ST.
I’m setting up a small NAS for the office (an SBC with an SSD connected), and installed on it are Openmediavault, a folder shared with Samba, and Docker with a linuxserver Syncthing container to sync files across all work computers. Thing is, it works fine on the LAN, but it won’t connect to relays, so it can’t sync when e.g. a laptop is working from another location. Other computers see the relays, so the issue seems limited to the containerized software only.
The logs show these repeating messages:
2021-12-08 07:10:56 Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
2021-12-08 07:10:56 Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down
2021-12-08 07:10:56 listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: service dynamic+https://relays.syncthing.net/endpoint failed: Get "https://relays.syncthing.net/endpoint": dial tcp: lookup relays.syncthing.net on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address
So it seems ST can’t reach the relay servers. I have checked:
in another location, on another modem/router, same result: PCs see relays, NAS doesn’t;
ping and nslookup can resolve into IPs and contact this and other domains (syncthing.net, google.com etc) both from the NAS’s Linux OS and from inside the Docker container;
ports 22000/TCP, 22000/UDP and 21027/UDP are open on the router;
netstat shows no other processes are using those ports;
both NAS’s and container’s etc/resolv.conf files have the same DNS servers in them, so they know where to “ask for directions”…
…but the Syncthing inside the container still can’t reach the relays. I’ve tried using network_mode: host but the error message just changes in:
2021-12-06 00:34:16 listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: service dynamic+https://relays.syncthing.net/endpoint failed: Get "https://relays.syncthing.net/endpoint": dial tcp: lookup relays.syncthing.net on [::1]:53: read udp [::1]:49236->[::1]:53: read: connection refused
Options: Nat traversal, Global discovery, Local discovery and Relaying, are checked and active in the GUI. I’m at my wits’ end. Is this a bug, or could there be another reason Syncthing inside a container can’t find/reach relay servers?
The host’s and container’s resolv.conf files have the same list of DNS servers, so I think I’m mounting the host’s one. How do I check to be sure?
But even if that was the case, since the NAS connects to the router with DHCP, the only DNSs listed in both resolv.conf are the same the router uses: a couple of external servers, 208.67.220.220 and 1.1.1.1 Nothing local.
I get that, I know that unless I use network_mode: host the container and host networks are separated; but why is Syncthing looking up localhost (and apparently only that) if resolv.conf specifies two external servers to resolve IPs? It can’t be that it’s falling back on localhost when it can’t find those DNS, the container can ping them.
I’m using linuxserver.io’s image without adding or changing anything, it should work no problem. I don’t get what’s going on. Is there something telling Syncthing to ignore resolv.conf…?
More or less the same. It pulled the image, started it as a brand new container, and then started looking up relays on localhost, not finding them and throwing “cannot assign requested address” errors on repeat (if I’m reading the logs right):
root@nas:~# docker run --pull always -it --rm syncthing/syncthing
latest: Pulling from syncthing/syncthing
9b3977197b4f: Pull complete
63dbf1db7fa8: Pull complete
bef99fb6914f: Pull complete
4dbdd16850e6: Pull complete
Digest: sha256:90e8aa1911b658e557967acd857e889b82eb64fa0ad7e93f0e4973a96ab55ad0
Status: Downloaded newer image for syncthing/syncthing:latest
[monitor] 23:33:08 INFO: We will skip creation of a default folder on first start
[start] 23:33:08 INFO: syncthing v1.18.5 "Fermium Flea" (go1.17.3 linux-arm64) docker@build.syncthing.net 2021-11-22 09:11:00 UTC [noupgrade]
[start] 23:33:08 INFO: Generating ECDSA key and certificate for syncthing...
[start] 23:33:08 INFO: Default folder created and/or linked to new config
[start] 23:33:08 INFO: Default config saved. Edit /var/syncthing/config/config.xml to taste (with Syncthing stopped) or use the GUI
[start] 23:33:08 INFO: Archiving a copy of old config file format at: /var/syncthing/config/config.xml.v0
[VQZUT] 23:33:08 INFO: My ID: VQZUTGO-UQUVLOA-HETXSC2-HROHO2H-W67CCMP-KPXPOQU-M2V3GAR-RUN4MAX
[VQZUT] 23:33:09 INFO: Single thread SHA256 performance is 527 MB/s using minio/sha256-simd (362 MB/s using crypto/sha256).
[VQZUT] 23:33:10 INFO: Hashing performance is 126.91 MB/s
[VQZUT] 23:33:10 INFO: Running database migration 1...
[VQZUT] 23:33:10 INFO: Running database migration 2...
[VQZUT] 23:33:10 INFO: Running database migration 3...
[VQZUT] 23:33:10 INFO: Running database migration 5...
[VQZUT] 23:33:10 INFO: Running database migration 6...
[VQZUT] 23:33:10 INFO: Running database migration 7...
[VQZUT] 23:33:10 INFO: Running database migration 9...
[VQZUT] 23:33:10 INFO: Running database migration 10...
[VQZUT] 23:33:10 INFO: Running database migration 11...
[VQZUT] 23:33:10 INFO: Running database migration 13...
[VQZUT] 23:33:10 INFO: Running database migration 14...
[VQZUT] 23:33:10 INFO: Running database migration 16...
[VQZUT] 23:33:10 INFO: Running database migration 17...
[VQZUT] 23:33:10 INFO: Running database migration 19...
[VQZUT] 23:33:10 INFO: Compacting database after migration...
[VQZUT] 23:33:10 INFO: Overall send rate is unlimited, receive rate is unlimited
[VQZUT] 23:33:10 INFO: No stored folder metadata for "default"; recalculating
[VQZUT] 23:33:10 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VQZUT] 23:33:10 INFO: TCP listener ([::]:22000) starting
[VQZUT] 23:33:10 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VQZUT] 23:33:10 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VQZUT] 23:33:10 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[VQZUT] 23:33:10 INFO: Loading HTTPS certificate: open /var/syncthing/config/https-cert.pem: no such file or directory
[VQZUT] 23:33:10 INFO: Creating new HTTPS certificate
[VQZUT] 23:33:10 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[VQZUT] 23:33:10 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
2021/12/17 23:33:10 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
[VQZUT] 23:33:10 INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
[VQZUT] 23:33:10 INFO: QUIC listener ([::]:22000) starting
[VQZUT] 23:33:10 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down
[VQZUT] 23:33:10 INFO: listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: service dynamic+https://relays.syncthing.net/endpoint failed: Get "https://relays.syncthing.net/endpoint": dial tcp: lookup relays.syncthing.net on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address
[VQZUT] 23:33:10 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VQZUT] 23:33:10 INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
[VQZUT] 23:33:10 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down
[VQZUT] 23:33:10 INFO: listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: service dynamic+https://relays.syncthing.net/endpoint failed: Get "https://relays.syncthing.net/endpoint": dial tcp: lookup relays.syncthing.net on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address
[VQZUT] 23:33:10 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VQZUT] 23:33:10 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down
[VQZUT] 23:33:10 INFO: listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: service dynamic+https://relays.syncthing.net/endpoint failed: Get "https://relays.syncthing.net/endpoint": dial tcp: lookup relays.syncthing.net on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address
[VQZUT] 23:33:10 INFO: GUI and API listening on [::]:8384
[VQZUT] 23:33:10 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[VQZUT] 23:33:10 INFO: My name is "228e1a5760c5"
[VQZUT] 23:33:24 INFO: Detected 1 NAT service
[VQZUT] 23:34:14 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VQZUT] 23:34:14 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down
[VQZUT] 23:34:14 INFO: listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: service dynamic+https://relays.syncthing.net/endpoint failed: Get "https://relays.syncthing.net/endpoint": dial tcp: lookup relays.syncthing.net on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address
[VQZUT] 23:34:14 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VQZUT] 23:34:14 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down
[VQZUT] 23:34:14 INFO: listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: service dynamic+https://relays.syncthing.net/endpoint failed: Get "https://relays.syncthing.net/endpoint": dial tcp: lookup relays.syncthing.net on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address
Apparently yes; (at first it was returning 127.0.0.11, but then I remembered to put back network_mode:bridge in the YML file) it’s the same result as using dig outside the container:
That’s why I came here: everything has the same settings, and I don’t understand if it’s Syncthing somehow not getting them or some other problem with the containers.
I keep a syncthing.yml file inside the docker folder, and launch the container with: docker-compose -f syncthing.yml up -d.
The config I’m using at the moment (I was tryin to strip it to the essential to narrow down the problem) is
Dockers default network driver (the default bridge) will by default spin up a forwarding DNS resolver reachable @ 127.0.0.11 : 53 (container’s network perspective). This DNS server will then be automatically configured in the container’s resolv.conf, so with the default bridge it should look like this:
This is done to avoid problems with the local systems resolv.conf*, as the network perspectives will be different. The container should send everything to docker, and docker will forward it to the host’s DNS servers.
While this works in theory, I’ve had sometimes trouble with Docker’s DNS server suddenly not responding anymore. I haven’t figured out what that was about, but my usual workaround for that is to add --dns "some external DNS server" to docker run (or equivalent for docker-compose) which will tell docker to put a custom DNS server in the container’s resolv.conf.
This however looks to me like Go is trying to talk to itself (container’s localhost IPv6), which is unlikely to work.
*This is not the only reason why docker does this. It also has a few other nice effects, such as being able to resolve container names in DNS and stuff.
That’s the part that drives me crazy. Everything looks like it’s configured right, I can ping and dig and DNS-lookup the same way from the host and from the container, but for some reason Syncthing keeps looking for relays on localhost. Seems like it can’t reach them, or it’s not reading the settings right. If there isn’t an option to force it by hand in using a specific route/IP, at this point I have half a mind to take an old image and try to reinstall Docker and ST from scratch, to see if anything changes.
Mind you, it’s entirely possible Audrius is right and it’s something in my whole setup, but I can’t fathom what issue can be so specific as to interfere in the working of a particular program inside a container in an openmediavault install where everything else is running as it should.
It’s not looking for relays on localhost. It tries to resolve the hostname “relays.syncthing.net” (where a list of all public relays is hosted) via DNS. This fails.
It’s surprising that dig is able to talk to configured DNS servers just fine while Go can’t do the same. But it’s also unusual that the container’s dig is not using the docker DNS service - that implies non-standard or manually modified configurations. What’s the exact content of your container’s /etc/resolv.conf?
I just had a look at linuxserver’s docker image and it seems they do some magic stuff with resolvconf, pulling the hosts resolvconf over, stripping out local servers and whatnot. This explains your config. No idea what’s wrong then.
At the moment, following Audrius’s experiment, I was actually using syncthing/syncthing image, but the result was the same, even trying to specify 8.8.8.8 as DNS server in the compose file.
I had a resolv.conf with 127.0.0.11, as you posted above, when I forgot to specify the network_mode, so I guess it used the defaults; at the moment, using network_mode:bridge, is like this:
docker exec -it syncthing cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 208.67.220.220
nameserver 1.1.1.1
That is, the exact same file with the same servers as the host, that in turn gets them from the router. If I set the DNS through the .YML compose file, it becomes just “nameserver 8.8.8.8”. It even shows up in a dig, but it still won’t resolve the relay.
I’m sorry, I’ve read the articles at that link, and tried setting the environment variable GODEBUG=netdns=go in my compose file, with no change. Maybe I’m missing your point, or I don’t know enough about Go. Should I set that env variable somewhere else?
I’m still confused. Tried with both yours and linuxserver’s image, tried setting it to go+2 or cgo+2 (starting without -d to see the messages in the terminal) and it always writes all the already-seen messages, plus
syncthing | go package net: built with netgo build tag; using Go's DNS resolver
in ANY case. I don’t get if it means it’s using some Go resolver without telling me which one, or it means it’s always using the internal Go DNS resolver every time, ignoring my setting and never using the C library resolver.