Syncthing Performance issues (Low speeds but 100% scalability?)

[Romans5.8 (John Hampton)]

Hello!

So my basic use case is this; I have a couple of SSD’s and a couple of RAID arrays on two machines which store lots of photos and videos (hobby photographer), to the tune of about 400GB on the SSD’s and about 600GB on an 8TB RAID0 array via USB 3.0 on the other machine. Nothing too crazy.

I have an older HP Workstation (Core2Duo 3.0GHz, 8GB of RAM, Windows 10 Pro) which has, in addition to its boot drive, two drives whose purpose in life is to exist as mirrors for that critical, fast storage upstairs. A 2TB spinning drive (to backup 2TB worth of SSD’s on Machine A), and an 8TB spinning drive (to backup the 8TB RAID 0 array on Machine B). Then, the HP workstation backs everything up using Crashplan.

Syncthing looked promising as a solution for getting all of the data, automatically, to the server. But the performance has been extraordinarily slow. And inconsistent. Sometimes running at 10-15mbps, sometimes down to less than 1mbps. (Small ‘b’, ‘megabit’, intentional)

At first, I thought maybe the old hardware was just the issue and I’d have to look for something that was better optimized for older hardware. (Or just live with it; as I have an upgrade planned in a few weeks). Except… it scales 100%. If I have syncthing running on BOTH machines at the same time; the speed effectively doubles. It still slows down, but when it’s running ‘full speed’, it hits 30-40mbps download on the server, with each client reporting 15-20mbps.

Is this the sort of speed I should be expecting? Are there ways to optimize it?

Everything is on gigabit ethernet. The two clients are a 2012 MacBook Pro (2.5GHz Core i5; GbE natively) and a 2016 Touchbar MacBook Pro (2.9GHz Core i7, GbE via a USB 3.0 adapter, LanSpeedTests gives me about 800mbps both directions to the server), and both should have sufficient CPU resources on their end. The client machines are set to ‘send only’ and the server is set to ‘receive only’.

Thoughts? Suggestions?

Thanks so much!!!

(Bonus question; is there a way to hide or ignore system files? I’m getting conflict errors which are all weird MacOS system files)

[AudriusButkevicius]

Check how they are connected (directly. Relay, etc). Syncthing will be slow with many smaller files, as there is overhead per file. You can setup a test folder with a single large file and see what results you get.

I’d also check io, cpu and ram usage on all machines, and benchmark network connections. It seems that the client side or network is the issue if doubling the machines providing the files doubles the speed.

Ignores are explained in the docs.

[Romans5.8 (John Hampton)]

Thanks!

I did figure out the ignores, works great! I just set up one; that ignores anything that starts with a period. Those are the same files macOS doesn’t display in the finder. I assume that’s probably all I need to ignore.

More than 24 hours after starting the initial sync, performance increased pretty significantly. More like 30mbps, definitely acceptable. I found that a few times, it would stop syncing completely and I would need to restart syncthing on the server.

IO, Network, RAM and CPU usage on all machines remained negligible, I never even saw the server CPU hit 10% which was especially surprising given it’s 10 year old dual core CPU.

As I said in the original post, I benchmarked network corrections and found speeds well within what I’d expect. And certainly significantly faster than what Syncthing was running. No relays were being used.

I’m not really sure what the issue was, but after several restarts when it would just stop syncing altogether, it finally sped up. Today I added a single 4GB test file to the directory and it synced at about 30mbps; which is much slower than if I were to transfer to the network share mounted on my desktop using the Finder, but it’s definitely an improvement over the performance for the first ~24 hours after I setup syncthing, and it’s in the category of ‘usable’ for my situation.