Discovery gives wrong IP (through docker)

Support
1

Hello, one of my devices is reporting or detecting the wrong IP address so that when my second device attempts discovery it gets the wrong IP.

I admit I have a bit of an odd arrangement:

Public machine “P” (device ID I7H4FLZ) is a server with multiple public IP addresses: 162.17.32.1-162.17.32.5. I am running syncthing in docker and connecting ports to 162.17.32.2. Actual command:

docker run --rm -h files.machinemotion.com -v /opt/syncthing:/opt/syncthing -p 127.0.0.1:8384:8384 -p 162.17.32.2:22000:22000 -p 162.17.32.2:21025:21025/udp --name syncthing utgllc/syncthing:latest

My laptop “L” (device ID KUMNZOX) is not publicly accessible. It’s just connected to a standard network with NAT and no uPnP, for instance at a coffee shop.

When I run syncthing on L with STTRACE=discover it seems to be getting the .1 address for P, not .2:

[KUMNZ] 2016/01/21 01:42:47.360523 main.go:564: INFO: syncthing v0.12.15 "Beryllium Bedbug" (go1.5.3 linux-amd64) unknown-user@build2.syncthing.net 2016-01-17 09:57:20 UTC
[KUMNZ] 2016/01/21 01:42:47.360634 main.go:565: INFO: My ID: KUMNZOX-OVFDDXA-LGE55AA-BZ5W7XL-U2AIDJE-GRYEHDL-YLAP2DM-5FG5FQF
[KUMNZ] 2016/01/21 01:42:47.678100 main.go:905: INFO: Single thread hash performance is ~184 MB/s
[KUMNZ] 2016/01/21 01:42:47.683293 model.go:192: OK: Ready to synchronize cmm-router-setup (read-write)
[KUMNZ] 2016/01/21 01:42:47.683351 main.go:752: INFO: Using discovery server https://discovery-v4-1.syncthing.net/?id=SR7AARM-TCBUZ5O-VFAXY4D-CECGSDE-3Q6IZ4G-XG7AH75-OBIXJQV-QJ6NLQA
[KUMNZ] 2016/01/21 01:42:47.683517 upnpservice.go:58: INFO: No UPnP device detected
[KUMNZ] 2016/01/21 01:42:47.683532 global.go:228: DEBUG: Announcement: {"direct":["tcp://0.0.0.0:22000"],"relays":null}
[KUMNZ] 2016/01/21 01:42:47.683543 main.go:752: INFO: Using discovery server https://discovery-v4-2.syncthing.net/?id=DVU36WY-H3LVZHW-E6LLFRE-YAFN5EL-HILWRYP-OC2M47J-Z4PE62Y-ADIBDQC
[KUMNZ] 2016/01/21 01:42:47.683618 main.go:752: INFO: Using discovery server https://discovery-v4-3.syncthing.net/?id=VK6HNJ3-VVMM66S-HRVWSCR-IXEHL2H-U4AQ4MW-UCPQBWX-J2L2UBK-NVZRDQZ
[KUMNZ] 2016/01/21 01:42:47.683688 global.go:228: DEBUG: Announcement: {"direct":["tcp://0.0.0.0:22000"],"relays":null}
[KUMNZ] 2016/01/21 01:42:47.683749 main.go:752: INFO: Using discovery server https://discovery-v6-1.syncthing.net/?id=SR7AARM-TCBUZ5O-VFAXY4D-CECGSDE-3Q6IZ4G-XG7AH75-OBIXJQV-QJ6NLQA
[KUMNZ] 2016/01/21 01:42:47.683837 main.go:752: INFO: Using discovery server https://discovery-v6-2.syncthing.net/?id=DVU36WY-H3LVZHW-E6LLFRE-YAFN5EL-HILWRYP-OC2M47J-Z4PE62Y-ADIBDQC
[KUMNZ] 2016/01/21 01:42:47.683925 global.go:228: DEBUG: Announcement: {"direct":["tcp://0.0.0.0:22000"],"relays":null}
[KUMNZ] 2016/01/21 01:42:47.684067 global.go:228: DEBUG: Announcement: {"direct":["tcp://0.0.0.0:22000"],"relays":null}
[KUMNZ] 2016/01/21 01:42:47.684114 main.go:752: INFO: Using discovery server https://discovery-v6-3.syncthing.net/?id=VK6HNJ3-VVMM66S-HRVWSCR-IXEHL2H-U4AQ4MW-UCPQBWX-J2L2UBK-NVZRDQZ
[KUMNZ] 2016/01/21 01:42:47.684242 global.go:228: DEBUG: Announcement: {"direct":["tcp://0.0.0.0:22000"],"relays":null}
[KUMNZ] 2016/01/21 01:42:47.684410 global.go:228: DEBUG: Announcement: {"direct":["tcp://0.0.0.0:22000"],"relays":null}
[KUMNZ] 2016/01/21 01:42:47.685257 rwfolder.go:332: INFO: Completed initial scan (rw) of folder cmm-router-setup
[KUMNZ] 2016/01/21 01:42:47.693251 main.go:802: INFO: Device KUMNZOX-OVFDDXA-LGE55AA-BZ5W7XL-U2AIDJE-GRYEHDL-YLAP2DM-5FG5FQF is "tsalmoth.serenevy.net" at [dynamic]
[KUMNZ] 2016/01/21 01:42:47.701362 gui.go:267: INFO: API listening on [::]:8384
[KUMNZ] 2016/01/21 01:42:47.701377 gui.go:268: INFO: GUI URL is http://127.0.0.1:8384/
[KUMNZ] 2016/01/21 01:42:47.717163 global.go:232: DEBUG: announce POST: Post https://discovery-v6-3.syncthing.net/: dial tcp [2400:6180:0:d0::d9:d001]:443: connect: network is unreachable
[KUMNZ] 2016/01/21 01:42:47.717186 global.go:232: DEBUG: announce POST: Post https://discovery-v6-2.syncthing.net/: dial tcp [2604:a880:800:10::182:a001]:443: connect: network is unreachable
[KUMNZ] 2016/01/21 01:42:47.717194 global.go:232: DEBUG: announce POST: Post https://discovery-v6-1.syncthing.net/: dial tcp [2001:470:28:4d6::5]:443: connect: network is unreachable
[KUMNZ] 2016/01/21 01:42:47.987285 global.go:237: DEBUG: announce POST: 204 No Content
[KUMNZ] 2016/01/21 01:42:47.987312 global.go:264: DEBUG: announce Reannounce-After: 1800 <nil>
[KUMNZ] 2016/01/21 01:42:48.260632 cache.go:107: DEBUG: lookup for I7H4FLZ-4TTOCRP-CXZ6KPI-QSAJ6XZ-YROSIJK-6DA7IPQ-TXEQH5T-IC32DQL at global@https://discovery-v4-1.syncthing.net/
[KUMNZ] 2016/01/21 01:42:48.260652 cache.go:108: DEBUG:   direct: [tcp://162.17.32.1:22000]
[KUMNZ] 2016/01/21 01:42:48.260659 cache.go:109: DEBUG:   relays: []
[KUMNZ] 2016/01/21 01:42:48.266561 global.go:237: DEBUG: announce POST: 204 No Content
[KUMNZ] 2016/01/21 01:42:48.266574 global.go:264: DEBUG: announce Reannounce-After: 1800 <nil>
[KUMNZ] 2016/01/21 01:42:48.501941 cache.go:107: DEBUG: lookup for I7H4FLZ-4TTOCRP-CXZ6KPI-QSAJ6XZ-YROSIJK-6DA7IPQ-TXEQH5T-IC32DQL at global@https://discovery-v4-2.syncthing.net/
[KUMNZ] 2016/01/21 01:42:48.501958 cache.go:108: DEBUG:   direct: [tcp://162.17.32.1:22000]
[KUMNZ] 2016/01/21 01:42:48.501962 cache.go:109: DEBUG:   relays: []
[KUMNZ] 2016/01/21 01:42:48.766537 global.go:237: DEBUG: announce POST: 204 No Content
[KUMNZ] 2016/01/21 01:42:48.766564 global.go:264: DEBUG: announce Reannounce-After: 1800 <nil>
[KUMNZ] 2016/01/21 01:42:49.643151 cache.go:107: DEBUG: lookup for I7H4FLZ-4TTOCRP-CXZ6KPI-QSAJ6XZ-YROSIJK-6DA7IPQ-TXEQH5T-IC32DQL at global@https://discovery-v4-3.syncthing.net/
[KUMNZ] 2016/01/21 01:42:49.643170 cache.go:108: DEBUG:   direct: [tcp://162.17.32.1:22000]
[KUMNZ] 2016/01/21 01:42:49.643175 cache.go:109: DEBUG:   relays: []

When I trace the discovery on the server P it does get the correct IP for my laptop L, but of course it is unable to reach it though the NAT.

I am able to reach the expected public port on P from L:

$ tcp_check 162.17.32.2 22000
162.17.32.2:22000 is accepting connections
$ tcp_check 162.17.32.1 22000
162.17.32.1:22000 is not accepting connections: Connection refused

I even edited the configuration on P trying to tell it its IP address (but I expect that device addresses are mainly for connecting to other devices, not for configuring oneself):

<!-- this device config is on the server P for the server P -->
<device id="I7H4FLZ-4TTOCRP-CXZ6KPI-QSAJ6XZ-YROSIJK-6DA7IPQ-TXEQH5T-IC32DQL" name="files.machinemotion.com" compression="metadata" introducer="false">
    <address>162.17.32.2</address>
</device>

I also have on P:

<listenAddress>tcp://0.0.0.0:22000</listenAddress>

but address/port control happens through the docker configuration which forwards connections to 162.17.32.2:22000 to the syncthing docker instance.

I did a discover trace on P, but the only line mentioning any of the IP addresses or announcements were these:

[I7H4F] 2016/01/21 02:00:12.001768 global.go:228: DEBUG: Announcement: {"direct":["tcp://0.0.0.0:22000"],"relays":null}
[I7H4F] 2016/01/21 02:00:12.016320 main.go:802: INFO: Device I7H4FLZ-4TTOCRP-CXZ6KPI-QSAJ6XZ-YROSIJK-6DA7IPQ-TXEQH5T-IC32DQL is "files.machinemotion.com" at [162.17.32.2]

Note: I can’t change listenAddress to tcp://162.17.32.2:22000 since syncthing in docker doesn’t have access to that interface, the connections are passed-through via whatever iptables magic docker does. (at least when I tried that syncthing immediately fails because it can’t bind to the that address.)

Any ideas what’s wrong or what I should look at next? Would any logs, configs, or other information be useful?

Thanks, Dean

2

Hi

I have no idea how to work around the issue of binding to an address from within a docker container. However I do wonder if the short term solution is to add this device to the cluster with a static IP?

It will at least get you up and running for now.

3

The discovery server uses your source address (because this is usually the right thing). Docker sets up incoming NAT but rewrites your outgoing traffic to the host address, like a regular NAT gateway. You’ll need to use static addresses, probably.

4

The IP detected by discovery is the IP that is used for making an outbound connection to discovery. I don’t think docker cares about outbound connections and it just uses whatever kernel decides is the default route.

A workaround could be adding a separate tun/tap device with that IP, and passing that to docker (not sure if its possible).

5

Ah, thank you all very much, that makes sense, and I’ve found success! A ServerFault thread: Use a specific outbound IP with Docker shows how to change the outbound IP. They are using:

iptables -t nat -I POSTROUTING -p all -s 172.17.0.0/16 -j SNAT --to-source 162.17.32.2

to change the outbound IP for all docker instances. I’d rather only change a single instance so I tried:

CONTAINER="$(docker ps -q -f name=syncthing)"
CONTAINER_IP="$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CONTAINER)"
iptables -t nat -I POSTROUTING -p all -s "$CONTAINER_IP"/32 -j SNAT --to-source 162.17.32.2

However, it didn’t work immediately (it very well may have if I were patient), but restarting the service caused docker to load another IP address so I didn’t really like that approach… At least for now. Apparently, docker has very recently merged some code for controlling addresses of containers (https://github.com/docker/docker/pull/16910), but it is too new for my system. So, for now I plan to hand-enter the server IP address on my client. Eventually when my docker gets the relevant update I’ll be able to assign syncthing a fixed internal IP and use the post-routing to give it the right outgoing IP.

Thanks for all your help!

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.