Starting the sync of large folder

[oregonjohn (John Statler)]

I am looking for help on starting the sync for 2 laptops and 4 home workstations with a folder on a server that contains 108k files totaling 310GB.

Throughout an average day 20 to 50 MB get added or updated to the 310GB folder. On exceptional days up to 1GB might get added or deleted, but that is planned to be done on Friday nights through the weekend.

The server, laptops and workstations are all running Linux Mint 17.3 and Syncthing v.012.22 with Syncthing-GTK. The server is connected to a 10Mb upload, 100Mb download WLAN. The others mostly have 1-2Mb upload and 10Mb download WLANs.

I see two possibilities for starting this sync, after which the computers will be taken to their remote location.

a) I could pre-seed the workstations with the folder either over the LAN or via USB drive. Once the initial transfer is done I would set up Syncthing to start it’s work.

b) I could just put all the computers I need to sync on the office LAN and start the sync from scratch without pre-seeding. The office LAN and all computers are gigabit ready.

From online calculators its seems that I will be lucky to get my 310GBs transferred out of the server hard drive at 70-100MB per second for a total transfer time over the LAN of about 70 minutes. If I have a bunch going at the same time it might be faster because Syncthing will be sharing among themselves, not just from the server (does that make sense?)

Now that I write that it does seem b) is the way to go but I would like some input.

I have read the manual 3 times and searched/read the forum numerous times trying to get a confidence about starting this sync – just so you know I’m asking this after having done some research.

I did try a few weeks ago and it seems I accidentally messed up the remote computer and it wound up deleting 50GB of data on the server. Fortunately I had a backup and no one had touched the folder between mess up and restore. Hard lesson learned, but I am looking for a way to identify when over 10 MB of data is being deleted and stop the process if that happens within a limited time frame.

Thanks to all for an awesome product.

John in Oregon

[calmh (Jakob Borg)]

You can do either and it’ll work. If you go down the preseeding route, make very sure to copy the index database that belongs to this exact set of files, or don’t copy it and all. Otherwise you’ll see deleted files as you mention.

From online calculators its seems that I will be lucky to get my 310GBs transferred out of the server hard drive at 70-100MB per second for a total transfer time over the LAN of about 70 minutes. If I have a bunch going at the same time it might be faster because Syncthing will be sharing among themselves, not just from the server (does that make sense?)

If your network is faster than 1Gbps and your disks support random writes faster than 100MB/s, and you have quite beefy CPUs everywhere, maybe… Don’t count on it.