discovery problems

Support
1

I am having problems with the discovery service. It looks to me like you got a partitioning problem.

This is a node where i just started a new service:

node1 $ curl -k "https://discovery.syncthing.net/v2/?device=E5XTCJP-QJLE4SR-OUNXQEY-VWRKXE4-ESJQ44R-CN6LFUQ-BLHXO3Q-P4Q5VA3" ; curl ifconfig.me; echo                                                        
{"seen":"2025-11-25T18:03:53Z","addresses":["quic://98.128.186.72:22000","tcp://98.128.186.72:22000"]}
98.128.186.72

However, when trying to do the same from another server:

node2 $ curl -k "https://discovery.syncthing.net/v2/?device=E5XTCJP-QJLE4SR-OUNXQEY-VWRKXE4-ESJQ44R-CN6LFUQ-BLHXO3Q-P4Q5VA3" ; curl ifconfig.me; echo                                                         
Not Found
91.145.2.73

I could do the same thing the other way:

node2 $ curl -k "https://discovery.syncthing.net/v2/?device=O62MN3M-OWV3KED-D2Y4CRD-77TA7WV-YTKOVYA-434RBX6-2FLS3PU-S3MK5QS" ; curl ifconfig.me ; echo
{"seen":"2025-11-25T18:16:45Z","addresses":["quic://10.51.82.2:22000","quic://192.168.1.200:22000","quic://91.145.2.73:22000","quic://[fd51:82f8:591a:5182::2]:22000","tcp://10.51.82.2:22000","tcp://192.168.1.200:22000","tcp://91.145.2.73:22000","tcp://[fd51:82f8:591a:5182::2]:22000"]}
91.145.2.73

Which won’t get picked up on the first node:

node1 $ $ curl -k "https://discovery.syncthing.net/v2/?device=O62MN3M-OWV3KED-D2Y4CRD-77TA7WV-YTKOVYA-434RBX6-2FLS3PU-S3MK5QS" ; curl ifconfig.me ; echo                                                        
Not Found
98.128.186.72

I don’t really know what kind of diagnostics could help right now (or if it is not a partitioning problem, what else kind of information would really help me or you), but this:

node1 $ dig +short discovery.syncthing.net                                                                                                                                                                      
94.130.221.203
94.130.164.126
157.90.0.173

node2 $ dig +short discovery.syncthing.net
5.9.87.175
65.109.78.228
88.198.34.105

and i could “force“ node2 to use one of the discovery servers from node1 and I will be able to se the same devices node1 sees:

node2 $ sudo tee -a /etc/hosts <<<"94.130.221.203 discovery.syncthing.net"                                                                                                                              
94.130.221.203 discovery.syncthing.net

$ curl -k "https://discovery.syncthing.net/v2/?device=O62MN3M-OWV3KED-D2Y4CRD-77TA7WV-YTKOVYA-434RBX6-2FLS3PU-S3MK5QS"
Not Found

node2 $ curl -k "https://discovery.syncthing.net/v2/?device=E5XTCJP-QJLE4SR-OUNXQEY-VWRKXE4-ESJQ44R-CN6LFUQ-BLHXO3Q-P4Q5VA3"  
{"seen":"2025-11-25T18:03:53Z","addresses":["quic://98.128.186.72:22000","tcp://98.128.186.72:22000"]}

Is this expected behaviour? To me it feels like this is kind of “wrong“, but what do I know? Both nodes are in the same small european country, but different ISPs.

2

That’s interesting and unexpected. The addresses in question are three nodes in the same cluster, and they should give identical answers under most circumstances. You can poll each of them by forcing curl to resolve to the IP directly:

% for addr in $(dig +short discovery-lookup.syncthing.net) ; do echo "From $addr:" ; curl --resolve discovery-lookup.syncthing.net:443:$addr https://discovery-lookup.syncthing.net/v2/?device=PSEUDOP-YLA5DBR-XO2HBNT-2FOG4AC-SWOW5EZ-H3QQY76-GISB7PG-E2SWOA2 ; done
From 94.130.164.126:
{"seen":"2025-11-26T17:23:36Z","addresses":["quic://10.1.37.64:22000","quic://172.16.32.4:22000","quic://83.233.124.224:22000","quic://[fd7a:115c:a1e0::5501:4880]:22000","tcp://10.1.37.64:22000","tcp://172.16.32.4:22000","tcp://83.233.124.224:22000","tcp://[fd7a:115c:a1e0::5501:4880]:22000"]}
From 94.130.221.203:
{"seen":"2025-11-26T17:23:36Z","addresses":["quic://10.1.37.64:22000","quic://172.16.32.4:22000","quic://83.233.124.224:22000","quic://[fd7a:115c:a1e0::5501:4880]:22000","tcp://10.1.37.64:22000","tcp://172.16.32.4:22000","tcp://83.233.124.224:22000","tcp://[fd7a:115c:a1e0::5501:4880]:22000"]}
From 157.90.0.173:
{"seen":"2025-11-26T17:23:36Z","addresses":["quic://10.1.37.64:22000","quic://172.16.32.4:22000","quic://83.233.124.224:22000","quic://[fd7a:115c:a1e0::5501:4880]:22000","tcp://10.1.37.64:22000","tcp://172.16.32.4:22000","tcp://83.233.124.224:22000","tcp://[fd7a:115c:a1e0::5501:4880]:22000"]}

(I’m not seeing the problem you see, and the metrics indicate they are in sync…)

1 Like
3

Yesterday, I tested @stuser51’s issue and was able to reproduce it using their Device IDs. I tested with my own Device IDs and was not able to reproduce it.

1 Like
4

Right so here’s the problem, “node2” is getting an outdated and incorrect DNS response. Those addresses haven’t been valid since November 14th. The servers backing them weren’t in the cluster, and were in fact destroyed the other day (so now will not respond at all). The TTL for these records was 3600 prior to the change and still is, so anything responding with those addresses after Nov 14 is broken somehow. (It could be that DigitalOcean, which still serves our DNS, had some sort out of problem/outage and briefly served old addresses…?)

3 Likes
5

After posting this I tried to get any other dns-provider than my ISP to send me the outdated records but couldn’t, and I noted that all six records where for the same German colo. I assumed you did some work around nov 15, so I changed to dns-servers to some public ones and everything has been working since.

Will file a support case towards my ISP.

I learned a lot about syncthing and the discovery process from this and I got to say you are doing a terrific job, real awesome stuff!

3 Likes
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.