I’m playing around with SyncThing and trying to understand how the file distribution between multiple devices works while devices are still syncing themselves.
In my setup, I have one device as “Send Only” (which we’ll call “Source”) and all others on “Receive Only”. My test case is adding a folder of about 48GiB size including 40k files to the source and measuring how long it takes until every device has received the data, so essentially optimizing for the last device to finish.
I have set the File Pull Order to “Alphabetic” to make it more likely that a slow device has multiple other devices to pull the file from it wants to sync next.
The different devices have varying latency and bandwidths to the source, with the worst being at 310ms RTT and about 80Mbps point-to-point transfer speed.
So my plan was that this device could pull from the source and also other devices that have already received the data, to mitigate the slow point-to-point speed between source and the worst target. That effect does occur, however it only occurs very early, and very late in the sync process of the full folder:
The diagram depicts the transfer speed in Mbps from each device to the slow target over the duration of the sync. The steady violet line is from the original source, and hovers around 80Mbps for most of the time, the expected point-to-point transfer speed. As you can see, in the beginning of the sync the target receives data from multiple other devices, capping at ~230Mbps in sum, and then (except for that one outlier in the middle) again at the end for ~377Mbps peak, when the other devices have already completed their sync.
For the majority of the time, devices other than the source aren’t uploading anything.
So my question is: Is there any mechanism that makes it less likely for a device to participate in the sync to other devices while it has not finished its own sync yet? Do you have any tips on how to optimize for the full sync to be completed across all devices as quickly as possible?
I think that effect might not work well for small files that are being pulled in the same order, as there is a latency between getting the file and others being aware that the device has the file. You’d be better off with random order.
Also given these are small files (by the virtue of how man there are), the sharing mechanisms might not kick in
You might have better results with random pull order or less much larger (gigabytes) files.
I had tried random order before, but I didn’t measure it as accurately. My impression was that the speed wasn’t great either. I will try that out again.
However, a weird observation is that if I actively pause certain connections, there is a significant speedup. I have limited the connections of the “far” device to the two devices it is closest to (the original source not being among them) and have seen a speedup of about 55% compared to a full mesh.
That means the devices do contribute to distribution while they are still syncing themselves, it’s just that the target does not make ideal decisions where to pull from (and in many cases will only keep pulling from the original source). I am sure that a setting in the form of “try to pull from at least N different devices” could result in an immense speedup in scenarios like mine.
If A B and C are connected, and B and C are aware that they both share the folder with A, for any given file they think they are missing, they will split the way they download the file.
So B will download first of the half, C will download second half, once they are done with their part, they will download the others part, at which point it should be available from more than one source.
This is obviously only meaningful in a single large file scenario.
Its done this way to reduce thrashing on the source, but obviously the sharing effect kicks in quite a bit later. 1/n way through the file where n is number of devices that participate.
There is a setting which you can set which makes the file part download order random, so the sharing effect might kick in earlier at the cost of thrashing.
In case of many smaller files, as long as the order is random, I feel it would achieve the same effect.
The decision on which device we send the request to is purely based on picking the device that has the least inflight requests (fast connections respond faster) and whether the device has what we are after (where what we are after is decided by file pull order or block pull order)
I don’t think “racing” requests from multiple sources makes sense, as it’s tcp, so even if you end up not wanting the data from the slow device anymore, it still has to arrive, causing head of line blocking for future requests, and consuming both of the devices bandwodth.
