Synchronising large files over slow networks with 10 devices

[hollymcr (Mark Rogers)]

I have 10 servers, many of which only have connection to the Internet over slow mobile links. As part of an upgrade I have several files I need share to all servers, totalling approx 2GB. They already have SyncThing installed and mutual sharing set up so that they all share a “Software” folder.

I don’t really care if it takes a couple of days to do this. What I am more interested in is not hogging the bandwidth on any server.

I don’t know all that much about the ST protocol. For example. if I put a large file into the folder on server 1, will it send different parts of that file to servers 2-10 so that they can all then get missing chunks from other servers, or will it essentially result in server 1 sending it to all 9 other servers?

What tips are there for making it more of a background process that gets there eventually without being noticed, rather than tying up the connections until the job is done (albeit more quickly)? Not least because my access to all these servers is via RDP over a VPN so if the connection is tied up sending files around I struggle to maintain access.

[gadget]

@hollymcr

I have 10 servers, many of which only have connection to the Internet over slow mobile links. As part of an upgrade I have several files I need share to all servers, totalling approx 2GB. They already have SyncThing installed and mutual sharing set up so that they all share a “Software” folder.

I don’t really care if it takes a couple of days to do this. What I am more interested in is not hogging the bandwidth on any server.

I don’t know all that much about the ST protocol. For example. if I put a large file into the folder on server 1, will it send different parts of that file to servers 2-10 so that they can all then get missing chunks from other servers, or will it essentially result in server 1 sending it to all 9 other servers?

Are all 10 servers interconnected in a mesh layout?

Or in other words, is each server’s Syncthing aware of its other 9 siblings? And the “Software” folder on each server is specifically shared with the others?

If so, then yes. Each server tries to find a source for the matching data block it’s looking for. The data block might be from an existing file or it could be downloaded from another node in the cluster. It’s not necessarily going to be evenly distributed due to network speeds, etc.

What tips are there for making it more of a background process that gets there eventually without being noticed, rather than tying up the connections until the job is done (albeit more quickly)? Not least because my access to all these servers is via RDP over a VPN so if the connection is tied up sending files around I struggle to maintain access.

There are a number of configuration tuning options in the web GUI and via Syncthing’s config.xml file.

In the web GUI, there’s a per-device rate limit under the “Advanced” settings for a device.