Hi,
we’d like to use Syncthing to synchronise several subsidiaries - across the world.
The problem is that they are really far away from each other (RRT about 200ms or even 400ms) - so that single tranfers over one TCP instance are really slow independant of the actual bandwith.
So, the question is how to speed up Syncthing in such a Long Fat Network Szenario?
Will it help to set up a bunch of private Relay Servers so that those are used to transfer files not only with one connection but with x connection (x= number of relay servers?)
Or are there other ways?
Thanks!
(If you’re not familiar with this problem, the keywords to check are LFN, BDP, TCP Windows Scaling and Buffer Sizes)
Syncthing does not support connecting to the same device more than once.
If you want to spend development effort, you could implement a new connection type that abstracts the fact that there are multiple backing connections.
We also have a UDP based solution coming some day, but I doubt it will solve problem as its all userspace.
What you can do, if you have enough HDD space and system resources, is to run multiple Syncthing instances in each subsidiary. They can run on the same machine.
I don’t know if it helps for big files, as both instances in one subsidiary could end up requesting the same block from the other subsidiary. But with temp index and local connection much faster than the WAN connection, that should be more the exception than the rule.
I use 2 Syncthing instances to speed up the Wifi transfer to my Android Phone, where the second connection helps a bit.
TCP transfers can be fairly fast even in high latency setups… Syncthing tried to set up large send and receive buffers, but in the end this is limited by the OS configuration. Ensuring that the buffer sizes and the TCP window size can get large is essential and tuning this from the OS site can make a difference.
On the Syncthing side you may need to increase the number of pullers to make sure we really fill the buffers and the TCP window. Essentially, the number of pullers is the max number of simultaneously outstanding block requests, per folder. The default of 16 pullers limits us to 16*128 KiB of data in flight at any given moment (per folder, but still). This needs to be set on the downloading side.
Hi Jakob,
I’ve been experimenting with TCP Settings for a while, and have no idea where to tweak TCP further, on Windows I’ve tested all the netsh setting (“autotuning=experimental”, where the RWIN can in theory get huge - but the real effect is barely measureable. Or “congestionprovider=ctcp” in send direction, this also has no real effect)
If you wan to Test it yourself:
https://jagt.github.io/clumsy/
The Tool Clumsy can simulate a high latency connection under Windows
On Linux you don’t need extra tools:
(i can’t post two links as a new user, just google
"tc: Adding simulated network latency to your Linux server")
About the Number of Pullers: I’ve tested 64 oder 128 in den config.xml - but there was no real difference with a transfer with a delay of ~ 200ms
I mean even when the TCP windows scales correctly
(I used Bitvise SSH Client to test this, because Syncthing is to messy to monitor with Wireshark) you get only 3-4Mbit with a 200ms connection - although you can see with wireshark that the Packets have a RWin Value of arount 2MB!
Okay, then this is out of the question because its a multi TB Project - and addtional Storeage costs will be higher then propriatry transfer solution for long distances (FileCatalyst, Apera etc.)