I am wondering if there is anything wrong with my setup. Basically, I have two synching endpoints. One sits on a Synology (DSM 7.1 with Docker) and the other one sits on a TrueNAS server. They are on two different subnets locally available at each location connected by an IPSEC vpn. Performance is extremely slow hovering around 1 MBit/s. The Syno location has around 900/40 MBit/s (download/upload speed) and the TrueNAS location has 100/100 MBit/s. I can actually use a filebrowser application on the TrueNAS and upload files at around 2.75 MBit/s. How can truenas be slower than a manual upload? Initially, one location showed as Relay WAN, but I fixed that by manually specifying the host name at each endpoint. For the port, I have put tcp://<syno_ip>:22000, dynamic for the TrueNAS endpoint and tcp://<truenas_ip>:20978, dynamic for the Syno endpoint. Both endpoint now show a connection type of TCP WAN. I am guessing it shows as WAN because they are different subnets. One is 192.168.x.0/24 and the other one is 10.10.y.0/24. Is 1 MBit/s all I can get or am I missing something in my setup. If it is, how can the manual upload be quicker? Any help is appreciated!
Although mathematically 2.75 Mbit/s is 2.75x faster than 1 Mbit/s, try not to get too distracted by it. Given your potential download/upload speeds, relatively speaking, both data rates are pretty subpar.
You didn’t mention how many files and typical file sizes are being synced by Syncthing, but with Syncthing running inside a container plus the normal overhead required for chunking + hashing + syncing, it can easily be chalked up to the 1.75 Mbit/s difference you’re seeing compared to copying with a plain file manager/browser.
I recommend putting aside Syncthing for the time being and focusing on the network setup itself. Even better, also ignore the Synology and TrueNAS until the IPsec VPN link between the two locations has been tested to see what kind of throughput to reasonably expect.
Once there’s a reference baseline for the IPsec VPN link, test the Synology and TrueNAS separately to rule out bottlenecks with either one.
Finally, put Syncthing back into the loop after having a good reference point to compare with.
@gadget Thank you for getting back to me. I was already planning on checking the connectivity and capacity of both devices today. I will also open up a question with regards to the vpn on Sophos/TrueNAS respectively. Once done, I will report back. I am looking at hundreds of gb to synchronize and while I don’t expect any wonders, I was hoping for roughly 10% of the available connection speed if things are idle. I am surprised to hear your comment about docker overhead. Should I try a native installation on the Syno. On the TrueNAS, I am already using TrueCharts and not Docker. Quite frankly, I am suspicious that the Syno is the bottleneck.
From my own experience with a variety of different VPN systems, it’s not unusual to get 90% or so of the raw bandwidth – i.e., transferring files from the Synology to the TrueNAS should be able to top out at more than 35 Mbit/s given the 900/40 Mbit/s available to the Synology.
Docker on Windows and macOS has significantly more overhead than the Linux based OSes for Synology and TruNAS.
(If you’re using FreeBSD, that’s a whole other discussion since the last time I checked, Docker Engine doesn’t run natively on FreeBSD.)
The reason Docker is more resource intensive on Windows and macOS is because they lack the required kernel components so Docker Engine is running inside a Linux virtual machine:
[Windows [Linux [Docker Engine [Container]]]
In contrast, Synology DSM and TrueNAS can run Docker Engine directly.
However, because Docker Engine is a daemon running with root privileges, Docker filters all system calls as a security measure to prevent processes inside the container from escaping the security sandbox (there’s a “rootless” mode, but that’s also for a different discussion).
Access to storage (bind mount vs. overlay) also has a significant impact.
CPU/RAM/disk I/O-intensive applications can run quite a bit slower in a Docker container compared to running bare-metal. However, real-world results depends on all of the conditions involved. My Plex server at home runs in a container (using Podman) and it has no problem streaming 2K video on a NAS over Wi-Fi with a dual-core CPU from 2010.
At the moment, there’s not enough info to definitely say where the bottleneck is. If you said that you’re syncing 2 million, 500 byte files from a HDD it would be very different from syncing a thousand 1GB files from a RAID0 of SSDs.
I don’t use MacOS now, but I wouldn’t run a Docker installation under Windows under any circumstances, there are excellent tools for that e.g.
I think that in addition to the data rates mentioned, IPsec VPN has a relatively high overhead, which in my experience is significantly higher as with Wireguard, for example. In this respect, all network components should be checked in this way, as already mentioned above.
Another effect will be that Syncthing will throttle, as the software takes the available bandwidth into account and does not want to block it completely on its own.
A big thank you to both of you for trying to help!
Yes, that is exactly why I moved syncthing from my Windows server and workstation to truenas and synology.
Here are some screenshots of both installations.
Synology NAS syncthing
That installation also has a seemingly unrelated issue.
The issue shown only happens for directional folders (e.g. send only).
The connected truenas
The folder status.
I am checking on the 6 out of sync items.
TRUENAS syncthing
That installation also has a seemingly unrelated issue.
Same folder as above.
The connected Synology.
The device name is not the same. It is shortened.
The folder status.
The third folder on the truenas is connected to another local installation running on Windows which has no issues.
The speed is now just 1 B/s on average… I am guessing it is either way idle or just can’t resolve some kind of connectivity or permission issue.
One question: What user is being used for the native syncthing install on Synology?
It’s been years since I worked with a Synology NAS so the default setting could have changed, but it looks like it’s still really low. Open a terminal on the Synology and check the Linux kernel setting for /proc/sys/fs/inotify/max_user_watches.
Whatever the number is, it must be large enough to handle the total number of files and directories being watched. Not only by Syncthing, but by all other processes running under the same user ID.
Thank you!
I have been able to resolve the inotify issue on both systems!
On the syno, I ended creating a boot task and, on the truenas, I maintained a configuration variable. Both solutions can be found in other posts on this forum with more details.
I have been able to resolve the permission problems and so all folders are synced now. Solution for permission problem is here.
The only issue that is left is performance, but I am not sure if the shown transfer speeds are correct because if they are, how could I have transferred like 700 gb in just a few days?
On the network side, I understand that ipsec isn’t ideal, but the sophos sg backend doesn’t support wireguard. I guess I could install a wireguard client on the truenas, but I am not sure if that will be faster or not.
I am open to suggestions!
With a 40 Mbit/s link, 700 GB would take just a couple of days, so not at all unusual.
You’re using the site-to-site VPN feature in Sophos Firewall?
If so, switch the VPN from IPsec to SSL (the latter is a simpler protocol). IPsec predates SSL, so it carries around some old baggage (e.g. 3DES, CBC).
From WireGuard’s performance page:
These benchmarks are old, crusty, and not super well conducted. In the intervening time, WireGuard and IPsec have both gotten faster, with WireGuard stil edging out IPsec in some cases due to its multi-threading, while OpenVPN remains extremely slow. It is a work in progress to replace the below benchmarks with newer data.
I use WireGuard at home and at work. At work, I’ve got access to OpenVPN, SSL VPN and WireGuard gateways, and I’ve found WireGuard to definitely be faster than the other two.
That being said, seriously consider whether or not tunneling Syncthing thru a VPN is really necessary. Syncthing already encrypts data in transit, so wrapping another layer of encryption adds a lot of extra overhead.
I don’t think the udm-pro can do ssl vpn. It offers ipsec, openvpn, wireguard and two proprietary formats named teleport and site magic. I chose ipsec because it is said to be quicker than openvpn.
I did use ssl site-to-site on the sophos in the past. In fact, I do have a RED 15 appliance lying around. Do you think that would be faster? I can try to get that to work.
OpenVPN is slower than IPsec and WireGuard, but not so much so that it’d limit your throughput to less than 2 Mbit/s.
Without knowing what the current network topology is like now, I could only make a wild guess as to if it’d be faster.
It’s unclear where the bottleneck is without a good reference point (i.e., raw bandwidth test with and without the VPN).
Two endpoints. One is Sophos SG (100/100) and the other one is Ubiquiti (900/40) UDM-PRO. I put the confirmed download and upload speeds in brackets for each endpoint. The speeds are actual non-VPN speeds! One is a 10.10.x.0/24 network and the other one is a 192.168.y.0/24 network.
For the Sophos SG, even the lowest tier desktop appliance is rated for 325 MBit/s over the VPN. I’d expect Ubiquiti to meet or exceed that (I’ve used their mesh Wi-Fi and EdgeRouter devices). So I think it’s safe to rule them out as potential bottlenecks.
The network speeds involved wouldn’t stress even cheap CAT5E UTP cable, but it’s still worth checking all of the local cabling on each end.
What’s the link between the two endpoints?
Have you benchmarked the link between the Sophos and Ubiquiti appliances using iperf3 or some other suitable tool?
All cables are 5E or better.
The Sophos sits in a hosted environment. So, 100/100 is the “guaranteed” speed under ideal conditions.
The UDM-PRO sits in a home office environment.
I will try to take some measurements in the next few days, but I can’t make any disruptive changes in the Sophos environment.