Slow sync rate on LAN

I’ve got two Dell servers on a LAN syncing at a rate of about 1.00 MiB/s. I’ve read through the FAQs and read quite a few posts on this subject, but have not found a solution.

The servers are cabled with CAT5 and connected with a Dell N2024 gigabit switch. The servers have plenty of RAM and CPU capacity. Each is set up with 4 x 1.2TB SAS 10K RPM 6GBPS 2.5" in RAID 10.

This is the folder being synced:

sync_folder

The server called Stella is the “source” where all the current files reside. They are being synced to the server called Tank, which started out with an empty folder. The sync rate hovers around 1.00 MiB/s (it’ll jump up to say 1.7 MiB/s then fall back to 800 KiB/s). When simply copying the same folder from Stella to Tank, the transfer rate is between 10-20 MB/s.

copy

Stella’s OS is Windows Server 2012 R2, Tank’s is Windows Server 2016.

Any ideas what can be done to speed this process up? This is a secure environment, so if disabling security features would speed it up, I’m fine with that.

Thanks in advance.

Scott

You seem to be syncing very large numbers of small files. In that case, it’s expected that Syncthing will be slower than other tools (which includes Windows Explorer seen in the screenshot). For why, I’d suggest reading the whole thread at https://github.com/syncthing/syncthing/issues/8602.

I think the only possible but not really practical method to speed things up would be running multiple instances of Syncthing, separating your folders between them.

Syncthing doesn’t really allow to disable any security features, so there’s nothing to tweak in this regard.

Thank you for the incredibly fast reply!

I get it. I’m all for reliability and correctness over speed.

Scott

1 Like

Also, CAT5 I think is 100Mbps, which is roughly 12MiB/s, so perhaps get a better CAT5e or CAT6 cable?

A proper CAT5 cable should handle a gigabit connection. But you can check your bandwidth with iperf3.

Why do you think that would help?

Network speed almost certainly is not the limiting factor here, but syncthing DB ops, FS sync or other per file overhead in syncthing. You can’t compare a simple copy operation with Syncthing syncing.

On what storage is the syncthing db located?

There’s no security setting to disable (nor would that help), just a bunch of safety (as in data) ones. The only might be recommendable is after setting the folder to receive-only, disable fsync until it’s synced once. Then re-enable it.

The simple copy test achieved transfer rates of 10-20 MB/s. Clearly, network bandwidth is not the limiting factor.

The Syncthing DB is located on the same RAID, different partition.

I didn’t expect Syncthing to match the copy speed. I’m aware there is overhead. I just didn’t expect the sync rate to take such a big hit considering the average file size is 500K (which I don’t consider “small”, but guess I should in this context).

I suppose when dealing with a large number of files, the best approach would be to get all devices as closely in sync as possible manually before introducing Syncthing.

Because I tested this.

  1. I shared one big folder with 45,000 files between Syncthing 1 and Syncthing 2. The synchronisation took 8:30 minutes.
  2. I split the folder in half, then shared subfolder 1 between Syncthing 1-1 and Syncthing 2-1, and subfolder 2 between Syncthing 1-2 and Syncthing 2-2. The synchronisation finished in 6:45 minutes, and I believe it could have been even faster, but my CPU was pegged at 100% during the whole process, as everything was using twice as much resources.

The DB should be on another drive, ideally on an SSD. A partition is only a logical separation.

3 Likes

Echoing @bt90, a lot hinges on database operations so Syncthing becomes very sensitive to database latency. Spinning disks are not the thing to have there, regardless of RPM and RAIDness. And as mentioned, a lot of operations per file (including multiple synchronous writes), so small files will be slow – again, especially on (comparatively) slow storage.

1 Like

Having the DB on another physical drive wouldn’t hurt, but I doubt it’s a significant factor.

I turned on all the logging facilities to see if that might shed some light on what’s going on. Below are log entries for a single file being synced. I removed the entries where it seemed to be moving along nicely.

2022/10/17 07:57:16.957415 set.go:281: DEBUG: glewa-wgnnf GetGlobal
... (log entries removed)
2022/10/17 07:57:17.763510 events.go:334: DEBUG: log 2049625 ItemFinished 
2022/10/17 07:57:20.268158 lowlevel.go:239: DEBUG: insert (local);

It takes about 800ms to get from GetGlobal to ItemFinished. Then there’s a 2.5 second delay before ‘insert (local)’.

Where does the doubt come from?
The logs are also consistent.
Also you get input here from people running syncthing and supporting users for years - always good to do your own thinking and be critical, but nevertheless it’s worth giving some weight to that. Btw. I have my db in memory, which I didn’t do out for fun (still there as it still works, even though I now can afford SSDs).

It just seems that with all the other time consuming work that must be done for each file (read from network, hash, write to storage) that a few DB queries would be relatively insignificant.

I’m in the process now of getting a benchmark for the number of files synced in an hour running on spinning disks. Then I’ll run it for an hour on a RAM disk.

1 Like

It’s a bit of both, but it adds up. That said, for most people who run into the “sync one million tiny files” situation, it happens approximately once per installation – on first sync. That’s not nothing, but it is a one time pain that will pass and which be worked around by doing the initial sync using simple file copy or similar and then letting Syncthing take over. Syncthing is certainly not the fastest tool if what you want to do is move lots of tiny files from A to B, but hopefully that’s not the long term use case.

Spinning disk vs. RAM disk test results -

For the RAM disk part of the test, I moved all the Syncthing runtime files, including the DB files to a RAM disk on both machines.

Files synced in one hour on spinning disks: 7487
Files synced in one hour on RAM disks: 8252

That’s better than a 10% performance gain! I expected a boost, but not that much.

Thanks to all the people who went out of their way to help me. I sincerely appreciate you all taking precious time out of your day.

This is a one-time-pain deal for sure.

1 Like

You might also want to try disabling fsync. It’s not safe in case the process is killed or loss of power, but gives another point of measurement.

I tried disabling fsync. Not much change. Actually slowed down a bit.

Cat5 is 100 and Cat5e is 1000. Cat5e degrading to 100 is a common problem (first-hand experience).

This is a good article. Near the bottom it explains a “failed” Cat5e or Cat6 cable.

[Ethernet Link Speed Capped at 100 Mbps]

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.