I’m in Hungary, yet after starting synthing three times, it chose three different Czech relays, while there are closer (both geographically and network wise) ones according to the relay list. Why is that? How does syncthing choose a relay server?
1 Like
It checks the latency of relays and puts them in 50ms buckets, trying the lowest-latency ones first. Which relay is chosen within those buckets is random.
1 Like
Thanks! So it probes all relays during startup. Is it possible to view this information?
BTW, for one of the czech servers ping looks like this:
--- 128.0.191.10 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9012ms
rtt min/avg/max/mdev = 21.264/23.800/25.014/1.094 ms
While for the two hungarian servers:
--- 195.228.252.133 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9010ms
rtt min/avg/max/mdev = 5.210/6.898/13.202/2.267 ms
--- 193.227.196.10 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9012ms
rtt min/avg/max/mdev = 11.333/13.028/15.285/1.232 ms
All are below 50 ms, so they will be in the same bucket and the random selection means the countries with the most servers will win (here the Czech, because it overweights all neighboring countries in the number of available relays and they are close network-wise). In understand the rationale behind the random selection in the buckets, it’s a wise choice (doesn’t need coordination and gives good balancing).
But 50 ms basically means whole Europe even from the eastern side (considering that most of the servers are in western Europe), so the farthest servers will overweigh the closest ones.
50 ms draws approximately a 5-8000 km radius on good networks, which means I will mostly communicate over international lines instead of local ones.
Could you please consider a smaller bucket size? Like 20 ms. I guess that could be a good tradeoff between too small buckets and the current long distance transfers.
Are you suggesting that crossing national borders is the real problem, rather than the not-quite optimal choice of servers?
Because there can easily be cases where the best servers are, in fact, across borders, even if that’s not true for you.
Well, it depends on whose problem is it. For the network operators, if the syncthing traffic would be a big chunk of the whole, the national/international thing would be the real problem.
For the end user (me), the real metric is the througput (optimal server selection). I thought it’s clear that I’m talking about that, sorry. 
To illustrate this, I’ve written a small script, which sends and receives data through syncthing’s testutil (which acts as a bidirectional pipe and can be fed/consumed from stdin/stdout, so seem to be fine for these kind of tests).
So I took my syncthing log from today and tested all the relays it has used. All speeds are MiBps (Megabytes per second) and I’m on a symmetric Gb residential connection. I placed the two hungarian (domestic) servers to the front, but please note that these were not selected by syncthing:
country | relay | top speed | avg speed
---------+-----------------+-----------+----------
HU | 195.228.252.133 | 29.24 | 27.26
HU | 193.227.196.10 | 26.17 | 11.38 (session-rate=10)
UA | 45.137.155.56 | 5.20 | 3.07
CZ | 128.0.191.10 | 2.45 | 1.50
CZ | 95.82.169.36 | 0.44 | 0.20
DE | 85.214.100.39 | 8.07 | 4.44 (global-rate=8.34, session-rate=3.81)
CZ | 185.8.166.21 | 0.31 | 0.13
NL | 51.15.105.255 | 14.30 | 11.55 (global-rate=30)
DE | 142.93.102.4 | 29.77 | 23.85
So syncthing has selected a relay 7 times (I’ve restarted it intentionally) and nearly all of them were hugely (one or two(!) orders of magnitude) slower than the domestic ones.
Well, we could call that not-quite optimal, yes. 
Relays is a measure of last resort. You should try to fix the reason why you can’t connect directly, not how the relay is chosen. I might start up a relay with 10kb/s limit next door to you and you’d still have the same problem, so the solution doesn’t make much sense.
2 Likes
Would the solution for this user be to use a direct connection using VPN or SSH?
It’s up to the user to decide what the solution is, I don’t think dropping latency to 25ms would fix this, as it could still pick a bad relay at 25ms.
The user can force syncthing to connect to a specific relay, if you have a strong preference, how to do that is explained in the docs.
Well, with the arbitrary limits in place, in the current scheme you are right. Using relays without any central coordination/balancing is a roll with the dice.
I still think decreasing the radius may give better results if relays tend to be well performing (like in this case, even the limited hungarian relay is mostly better than the chosen ones), but I don’t have any data which could support this of course…
Direct connections don’t work here (double NATs, corporate firewalls etc).
So this isn’t in focus, I understood. Thanks for answering and for the great product!
1 Like
In theory Syncthing could use several relays simultaneously, say up to 5, and periodically remove the worst of them to try a new one.
Sure, and then you have to implement tcp in software to stitch multiple different data streams with different latencies to have guaranteed in order delivery, handle failure etc. It’s much harder said than done.
Again, relays are last resort to get something connected, there are no guarantees around their performance.
As Syncthing only makes one connection to a device, it cannot use several relay servers simultaneously.
Relays might be “the last resort” from the perspective of an ideal world, but they are quite often used: Right now there are ~16’000 connections over relays of about 70’000 active users. So it is well worth to review how relays are chosen. It is out of question that the current algorithm to chose a relay is too simple to get the best performance possible for each user.
On our relay, which is located in Zurich/Switzerland, we see the following distribution of connections per country [1]:
226 TH, Thailand
142 DE, Germany
138 CN, China
70 US, United States
68 FR, France
55 RU, Russian Federation
44 GB, United Kingdom
36 CH, Switzerland
28 NL, Netherlands
21 CZ, Czech Republic
20 SE, Sweden
20 IT, Italy
18 AT, Austria
16 IP Address not found
15 VN, Vietnam
14 UA, Ukraine
14 PL, Poland
12 JP, Japan
11 IR, Iran, Islamic Republic of
11 GR, Greece
11 AU, Australia
10 IN, India
9 ZA, South Africa
9 DK, Denmark
8 IL, Israel
8 ID, Indonesia
8 HR, Croatia
8 BR, Brazil
7 HK, Hong Kong
7 CO, Colombia
6 RO, Romania
6 PT, Portugal
6 NO, Norway
6 HU, Hungary
6 ES, Spain
6 CA, Canada
6 BE, Belgium
5 SG, Singapore
5 FI, Finland
4 SA, Saudi Arabia
4 PA, Panama
4 IE, Ireland
4 BG, Bulgaria
3 AR, Argentina
3 AO, Angola
2 TW, Taiwan
2 SI, Slovenia
2 MY, Malaysia
2 MX, Mexico
2 LV, Latvia
2 EU, Europe
2 EC, Ecuador
1 VA, Holy See (Vatican City State)
1 TN, Tunisia
1 SK, Slovakia
1 RS, Serbia
1 PK, Pakistan
1 PH, Philippines
1 NA, Namibia
1 LT, Lithuania
1 KW, Kuwait
1 KR, Korea, Republic of
1 JO, Jordan
1 GG, Guernsey
1 EE, Estonia
1 DO, Dominican Republic
1 BY, Belarus
1 BW, Botswana
1 BA, Bosnia and Herzegovina
Let’s look at those 70 connection form the US: Ping times from Switzerland to the US are always over 85ms. There are enough relays in the US that have lower latency. For these users the choice to use our relay is certainly not ideal. For the japanese and australian users the choice of a european server must be really frustrating!
There are user complaints about the choice of relays and the data bra delivers draws a clear picture. To decline the problem with the “last resort” reason is not the open minded attitude a developer should show towards proven real world problems. So please, let’s discuss the issue.
I think what hiq suggests is not having simultaneous connections to several relays, he rather suggest “trying” different candidates, who seem suitable, by using them for a while and rate their real world performance. Then switch to the one who performed best. This is probably difficult to implement, because bandwidth only really matters if the work load to transfer is bigger and takes a while. A user wouldn’t understand why a first file goes fast and the second a few minutes or seconds later is much slower transferred than before. The measurement depends on daytime, load on the relay, state of the network etc., loads of changing parameters. I think this would be quite a task to implement a well working solution.
The suggestion from bra to reduce bucket threshold latency is difficult for users who live in an area with just a few or no relays or where network latency is generally a problem.
Based on that suggestion you could introduce a dynamic bucket threshold latency: Start with 10ms, fill the bucket with candidates, if there are less than 5, double bucket threshold latency, fill, double and so forth until you have 5 candidates in the bucket. You could also reduce the needed number of candidates with each doubling by 1 (but not fewer than 1 obviously).
This is a minimal change and will prefer low latency connections and works also for users with slow networks or sparse relay distribution.
The problem that bandwidth and latency is not the same is not solved with that approach. Real bandwidth can only be probed, but we don’t want to waste bandwidth with probes. So it will still be an estimate.
Administrators of relays know quite well how much bandwidth they can supply. They give a hint by setting the global and per client limit. Furthermore the number of connections/sessions and current load is reported by the relays. If one would combine those numbers to deselect a relay from the candidate, the estimate of good throughput could improve.
Thanks for not denying this discussion, even if relays are not the best option for a user, relaying is important for many of us not only because of firewall/NAT restrictions, also because you cannot expect from an average user to open port 22000 nor does that solution work for multiple users behind the same NAT.
Regards, Adrian.
[1] Quick and dirty, on the relay do: RELAY_IP=X.X.X.X ; while read -r ip ; do geoiplookup “$ip”; done <<< $(netstat -n | grep “${RELAY_IP}:22067” | sed -e “s/^.:22067[^0-9]//g” -e “s/:.*$//g”) | sort | uniq -c | sort -rn | sed -e “s/GeoIP Country Edition: //g”
1 Like
Thanks for providing a relay!
The statement “relays are a matter of last resort” is not to shut down any discussion, it’s a fact about a design choice: Syncthing tries to connect whenever possible, and goes quite some length to achieve that. The last measure taken is connecting by relay, and thus it’s a last resort. That doesn’t mean that relaying can’t/shouldn’t be improved. You’ll see a lot of past discussion on it, there just wasn’t anyone (yet) interested in following those up with implementations. PRs are welcome.
You know, funnily enough, that’s the underlying issue here. If latency and bandwidth were more correlated, a more selective latency selection would be good.
It’s just that, well, they aren’t.
That mainly doesn’t work because conventional wisdom on NAT is influenced by idiots who believe it’s a security solution.
@odin: There is relation of latency and bandwidth: If latency is very high, bandwidth won’t, at least not from beginning. For smaller files low latency is needed to have higher throughput. They are enough correlated to do better than syncthing does at the moment.
@imsodin: OK, I got that with “last resort”, it sounds much better like this. It would be worth a try with “dynamic bucket threshold latency” in somewhat the way I have described it to improve syncthing’s chosing of a relay and quite easy to code. Where would be the place to inspire a developer to implement it? I just do bash, that’s not what’s needed, I guess…
@Imsodin: I don’t understand the code really, but from what I understand I think you could just make the buckets smaller in /syncthing/tree/main/lib/relay/client/dynamic.go, line 159:
id := int(latency/time.Millisecond) / 20
(change divisor from 50 to 20)
If I interpret the code right it doesn’t care for empty buckets. The function relayAddressesOrder does return all answering addresses that were tested just reordered, right? So changing the size of the buckets from 50ms to 20ms buckets doesn’t really hurt, it might eliminates some randomness (but only if there are not enough relays in a bucket).
The implication would be in areas with dense relay distribution that relays with lower latency would be chosen with a higher chance, which is what @bra whished (the code already assumes a positive relation between latency and bandwidth and for smaller files this is certainly true).
The implication in areas with a sparse relay distribution is uncertain: It could well be that there is a concentration on the few relays there are and they might get overloaded – but this remains to be seen. But instead of experimenting with the user’s and relay owner’s patience, we can do better…
A very easy improvement would be to make the bucket size growing with higher latencies. This would give back the randomness for the sparse distribution areas and keep the low latency choice for dense areas:
id := int(Sqrt(latency/time.Millisecond / 10))
(I’ve never written anything in golang, so regard this as pseudo code that means: “Divide the latency in ms by 10, take the square root and round down”).
This gives the following buckets:
ID latency range
0 0 - 9 ms
1 10 - 39 ms
2 40 - 89 ms
3 90 - 159 ms
4 160 - 249 ms
5 250 - 359 ms
...
The suggested buckets seem much more real world than just equal 50ms sized buckets.
Calculating the square root over all addresses doesn’t take a lot of time compared to osutil.GetLatencyForURL(ctx, relay), so I think this is ok to introduce and does not give a huge delay.
What do you think?
PS: What is this stupid error from the forum software “sorry new users can only put 2 links in posts” good for? If a link is within syncthing’s domain or its code repo, it’s a useless protection and just annoying.
1 Like
Since I asked to continue this discussion on the forum - I think this doesn’t take into account that people don’t have a zero latency connection to the internet. Perhaps you mean to normalise by subtracting the minimum measured latency from everyone? If so, is the distinction relevant if you had a 100 ms base latency? I’m not so sure.
Besides, there are other possibilities for improvement. Perhaps it would be simpler and better if the relay pool server used geolocation to return only the 50 relays closest to the client? Perhaps it should select a subset based on connection load in order to try to even it out?
I now remember why we moved from 20ms to 50ms.
That’s because the number of relays in the bucket was sometimes one, and because we pick a single bucket and cycle through the relays as they fail, we’d end up with a bucket of a single relay and never connect anywhere else if its bad
You also don’t want to pick all the relays as that’s clearly bad too.
Square root doesn’t help with this and brings this small bucket one relay problem back.
Firstly: I am not convinced that more fine-grained latency selection will result in better relay connections. I mean the info about connection location from @phloggu is certainly interesting, however it doesn’t suggest latency being the problem to me:
Why do they then select that relay? The 50ms bucket should separate those 85ms latency connections out fine. Another explanation might be that a “local” client established a connection with it, announced that to global discovery and a “remote” (US or whatever) established a connection there - no latency goes into that consideration.
Basically I don’t want to think about solutions and implementations unless there’s a clear indication the latency bucket size is really a problem, or conversely changing it will improve things.
Having more finegrained latency buckets and still retaining enough relays per bucket would be simple: Have smaller latency buckets and define a min number of relays per bucket. Then merge adjacent buckets if they don’t reach the min.
2 Likes