Curious about sync efficiency /protocol for newly added files

General
1

I have syncthing deployed on approximately 30 servers at the moment to keep a common ‘deployment’ folder up-to-date. Each of those 30 servers are at their own separate office. They are connected via VPN tunnels, and they only sync on the private network.

The system works well, but I’ve noticed something about syncing new files that appears to be inefficient.

When I drop a file in to our deployment folder at the main site, the sync takes about the same amount of time to complete at every site. Download speeds are unrestricted, but upload speeds are restricted to 2 mbit/sec.

The amount of time it takes appears to be the amount of time it would take for me to manually copy the file to all 29 other locations at a rate of ~2 mbit/sec.

It seems like syncthing takes a chunk of the file and then feeds it to each of the other 29 servers. Then it takes the next chunk and feeds it…then the next until all 29 servers have received a complete copy of the file from the main server. This means the main site distributes a file, it’s doing all the work of uploading and distributing.

Maybe my non-scientific observations are wrong.

If not though, wouldn’t it be more efficient to chunk up the file and send it ‘round robin’ and let nodes download from each other?

Chunk 1 goes to Server 1, Chunk 2 goes to Server 2, etc…until at least one complete copy of the file exists on the network…and then the server can send Chunk 2 to Server 1, Chunk 3 to Server 2, etc…

The moment a server receives a chunk, other servers can download it from them instead of the master.

Bringing a new location online is obviously faster and more efficient–the new location connects to all 29 servers and receives chunks as fast as they can be sent.

Anyways–not a support issue or a feature request really. Just curious if I’m correct on how the sync works.

2

Kinda, sorta. What actually happens is that the main site sends an announcement “I have this file now, consisting of these blocks” to all its 29 peers. They will then prepare to “pull” that file, with the blocks in random order. For each block they’ll check which other device has announced availability of those blocks – to begin with, there will be only one. Over time, devices will announce blocks from partially downloaded files and be able to help each other out.

One issue here might be that “over time” can be quite slow and that the decision where to download blocks from is taken quite a while in advance, as multiple requests are queued at the same time. For example, if the file is smaller than 64 MiB all of it might be requested at once, and the only place available to request from in the beginning is the original source… If your file is gigabytes in size I would expect good load sharing between devices once they have some of the data.

3 Likes
3

Interesting–thanks! I have a few ISOs I need to shove into the deployment folder. I’ll play around and see what I can come up with. Since most of the ISOs are publicly available (FreeBSD, UBCD, etc…) I think I’m also going to try simultaneously moving them into the folder on 2 or 3 of the servers at the same time and see how well it works.

There’s definitely good load-sharing when I bring a new location online. It downloads chunks from every other server and it’s finished in no time. ~30 servers * 2 mbit/sec upload = ~60 mbit/sec download.

Thanks again for the insight.