Hello, first thanks for making and supporting such an awesome piece of software.
I am using this software to keep files between my home and office synched. I have everything setup and the process appears to be working, however the transfer speeds are very slow. Rarely above 1mb/s. AS my office has symterical 2k/2k fiber and my house has 400/400, I would think it should be much faster. The office and house are connected by a site to site wireguard VPN tunnel so both devices are essentially on the same network. The source end is a container running on unraid, while the house end is a windows machine with a synology nas connected to it via a 10G switch. The scans are also taking a very long time, but I just set this up yesterday.
While the bird’s eye view of the hardware and network connections is helpful, there are a lot of important details missing that prevents a proper diagnosis:
How many files and directories are being synced?
What kind of hardware is hosting Unraid?
What kind of hardware is the Synology NAS?
How does the Synology NAS figure into the setup? (e.g. Is the Windows machine running Syncthing or is the Synology? If it’s Windows, is it syncing to a network share?, etc.)
Although the site-to-site WireGuard tunnel connects the two endpoints, it won’t necessarily appear as the same network from a network routing perspective depending on the particular configuration.
Screenshots of Syncthing’s web GUI from both endpoints with folder and device details visible would be most helpful.
(I’m sure there are more details that would help, but those are the ones that immediately come to mind.)
Given that the scans are taking a long time, there are likely lots of files and/or insufficient hardware involved.
(Office) Unraid Source end is a dell R730 Enterprise server with 48 cores, 196GB RAM, and currently showing less than 20% utilization of the resources.
Target end is a small nuc style computer running an intel xeon E3 @ 3.2GHZ and 32GB ram. The only purpose of this computer is to run homebridge for my house automation, and Synchthing which is using the synogogy shares via mapped network drives.
It should be noted that the unraid server is also synching to another local machine in the same building for redundancy. Those shares are designated by “S” and are working fine.
I am attempting to clone 2 entire drives. T Drive is:
A good starting point is to first get a rough idea of the available network throughput between the Unraid server and the “HomebridgePC” thru the WireGuard VPN. Ideally, use a network diagnostic tool such as iperf3. Be sure to run the benchmark in both directions.
On paper, a symmetric fiber link at the office + relatively high speed broadband at home + a 10G network switch for the home LAN + well above average server and workstation hardware for the Syncthing endpoints says transfer speeds should be easily 300Mbps, but the SMB storage backend (a guess, based on the use of Windows on the HomebridgePC), VPN tunnel and other components create a number of potential bottlenecks, including:
Syncthing chunks and hashes files as part of its block deduplication feature, so every file in the Syncthing folder on the network share between the Synology NAS and HomebridgePC must be transferred twice during every file update, once to read the file plus once again to write the new version.
The Synology DS1821+ has 4 gigabit Ethernet ports. Unless two or more ports are being bonded, 1Gbps (or closer to 900Mbps or less in actual use) is the max speed. Factor in the overhead from the SMB protocol and it’s likely to be a fraction of that. (Although not a sure thing, disk I/O shouldn’t be an issue given the hardware list.)
On the Unraid side, Syncthing is running inside a container (you didn’t mention anything about it, but I’m guessing it’s Docker). Although the Dell R730 was showing less than 20% resource usage, because a container is similar in some ways to a virtual machine, how much of the 20% is attributed to Syncthing? For example, if the container is limited to 1 CPU core, 1 GB of RAM and a 100Mbps NIC, Syncthing isn’t going to be able to move 1.75+ million files across 75K+ directories very quickly.
On the HomebridgePC, is there any antivirus software running? Does it screen network traffic?
There are more things to check, but first rule out the network link itself.
I’m a bit confused. You mention a server in your office but a NUC and a NAS on your side. Are they both running Syncthing or are you mounting a NAS folder on that NUC?
Ok, so I ran iperf3 from both sides and I think the tunnel might be the issue, I’ll fool around with it and try it without the tunnel over the weekend. Also, the NAS has another dual 10G nic card installed (the 4 Ethernet ports aren’t used) that are bonded and connected directly to the 10G switch with DACs. Yes on unraid it is running in a container. I don’t have anything pinned so the container has access to all the resources. Currently sitting at 9% usage. No antivirus software on either machine at the moment other than the stock windows defender BS on the homebridge machine.
Office side:
Ok, a little more of a wrinkle. I just realized my primary wan connection is down at my office, so I’m running on my backup which only has a 1000/40 speed. I’m sitting at home on my wireless iPad running Iperf and I’m getting the expected speeds over the WireGuard vpn bridge. I’m going to try and install SYNC directly on the NAS to rule out the SMB share slowing it down. Do you think that is a good idea?
You’re welcome. By the way, as a convenience, iperf3 has a -R option for reversing the transfer direction (default is for the client to send/upload to the server).
In your screenshots, when the Unraid box is uploading to the HomebridgePC, there were 55 retries, but in the reverse direction there were none. That’s a lot of retries for less than 20MB of data. Try extending the length of the test to see if the errors increase. For example, a 60-second test with status every 5 seconds:
Under each remote device setting, under the “Advanced” tab, there’s an “Addresses” box that defaults to “dynamic”. You could hard-code the endpoint WAN addresses to force the use of a particular network interface.
For most users, “dynamic” probably works just fine. However, mixing in a VPN, containers, bridged networks, etc. sometimes requires manual settings (fyi… I interchangeably use WireGuard plus a FortiGate VPN and haven’t had an issue with Syncthing finding the most efficient network path – it finds my NAS both on the LAN and when I’m at a random public hotspot).
Although the SMB share adds considerable network I/O, given your hardware setup, I can’t see it causing the transfer speeds to hover at 14-16 Mbps.
If you’re running WireGuard for iOS, given the iperf results, it’s worth looking closer at your HomebridgePC that’s running Windows because iOS <-> Unraid didn’t encounter any network retries and had respectable transfer speeds (although still low given the hardware and network connectivity).
With iOS test, my tunnel is site to site. If I’m connected to my network at home, I have access to my office from any device. No additional WireGuard needed on the iPad.
I just ran Iperf tests between the office and my house from a different hardwired machine in my home. Same results with the errors on the unraid machine upload. Must be something in my home network that is only affected the wired devices as it worked great on the iPad.
Given the good iperf results between the iPad and Unraid, it seems more likely to be a problem on the home side (specifically the HomebridgePC and/or its wired network connection).
I just plugged the homebridge machine directly into the router, same result. This is weird.
Homebridge is dac into the 10g.
My main gaming rig is a fiber run directly into the Sfp port on the router. I’ll try a new patch cable between the ont and the router tomorrow.
So I figured out the issue. My ISP had me double NAT. I arranged to get a static IP and the issue have resolved. I route around the WG tunnel as i get better xfer rates. Probably due to a MTU setting on the tunnel, but it works great once I avoid it.
