I’ve got two Dell servers on a LAN syncing at a rate of about 1.00 MiB/s. I’ve read through the FAQs and read quite a few posts on this subject, but have not found a solution.
The servers are cabled with CAT5 and connected with a Dell N2024 gigabit switch. The servers have plenty of RAM and CPU capacity. Each is set up with 4 x 1.2TB SAS 10K RPM 6GBPS 2.5" in RAID 10.
This is the folder being synced:
The server called Stella is the “source” where all the current files reside. They are being synced to the server called Tank, which started out with an empty folder. The sync rate hovers around 1.00 MiB/s (it’ll jump up to say 1.7 MiB/s then fall back to 800 KiB/s). When simply copying the same folder from Stella to Tank, the transfer rate is between 10-20 MB/s.
Stella’s OS is Windows Server 2012 R2, Tank’s is Windows Server 2016.
You seem to be syncing very large numbers of small files. In that case, it’s expected that Syncthing will be slower than other tools (which includes Windows Explorer seen in the screenshot). For why, I’d suggest reading the whole thread at https://github.com/syncthing/syncthing/issues/8602.
I think the only possible but not really practical method to speed things up would be running multiple instances of Syncthing, separating your folders between them.
Syncthing doesn’t really allow to disable any security features, so there’s nothing to tweak in this regard.
Network speed almost certainly is not the limiting factor here, but syncthing DB ops, FS sync or other per file overhead in syncthing. You can’t compare a simple copy operation with Syncthing syncing.
On what storage is the syncthing db located?
There’s no security setting to disable (nor would that help), just a bunch of safety (as in data) ones. The only might be recommendable is after setting the folder to receive-only, disable fsync until it’s synced once. Then re-enable it.
The simple copy test achieved transfer rates of 10-20 MB/s. Clearly, network bandwidth is not the limiting factor.
The Syncthing DB is located on the same RAID, different partition.
I didn’t expect Syncthing to match the copy speed. I’m aware there is overhead. I just didn’t expect the sync rate to take such a big hit considering the average file size is 500K (which I don’t consider “small”, but guess I should in this context).
I suppose when dealing with a large number of files, the best approach would be to get all devices as closely in sync as possible manually before introducing Syncthing.
I shared one big folder with 45,000 files between Syncthing 1 and Syncthing 2. The synchronisation took 8:30 minutes.
I split the folder in half, then shared subfolder 1 between Syncthing 1-1 and Syncthing 2-1, and subfolder 2 between Syncthing 1-2 and Syncthing 2-2. The synchronisation finished in 6:45 minutes, and I believe it could have been even faster, but my CPU was pegged at 100% during the whole process, as everything was using twice as much resources.
Echoing @bt90, a lot hinges on database operations so Syncthing becomes very sensitive to database latency. Spinning disks are not the thing to have there, regardless of RPM and RAIDness. And as mentioned, a lot of operations per file (including multiple synchronous writes), so small files will be slow – again, especially on (comparatively) slow storage.
Having the DB on another physical drive wouldn’t hurt, but I doubt it’s a significant factor.
I turned on all the logging facilities to see if that might shed some light on what’s going on. Below are log entries for a single file being synced. I removed the entries where it seemed to be moving along nicely.
Where does the doubt come from?
The logs are also consistent.
Also you get input here from people running syncthing and supporting users for years - always good to do your own thinking and be critical, but nevertheless it’s worth giving some weight to that. Btw. I have my db in memory, which I didn’t do out for fun (still there as it still works, even though I now can afford SSDs).
It’s a bit of both, but it adds up. That said, for most people who run into the “sync one million tiny files” situation, it happens approximately once per installation – on first sync. That’s not nothing, but it is a one time pain that will pass and which be worked around by doing the initial sync using simple file copy or similar and then letting Syncthing take over. Syncthing is certainly not the fastest tool if what you want to do is move lots of tiny files from A to B, but hopefully that’s not the long term use case.