Why transfer speed is so slow ? I get an average of 5 MB/s on a gigabit connection. The same behavior using freefilesync from an external Windows machine runs at 70~100 MB/s. I know this is not the only thread to talk about this subject. But my situation is quite far away of all I have read in all threads.
I did configure syncthing on 2 Synology NASes (amd64), both on the same LAN in gigabit. Both NASes handle full speed transfers on SMB is up to 110 MB/s, with an average of far above 50 MB/s in SMB. Most files are some big video files of 1GB or more each, with a total of 10TB. Syncthing is configured for 1-way syncing (RO -> RW).
I’ve checked all what was suggested in other threads, and other points:
CPU - < 10% on each NAS
RAM - < 20% on each NAS (src: 1/8G, dst: 1/4G)
load - < 2 on the source (4 cores), ~4 on the destination (4 cores) - the destination has higher I/O idle (SMB brings the load up to 8, with very good transfer speed)
scanning / hashing - not related - hashing was finished more than 2 weeks ago
crypting / hashing because of slow CPUs - not related, both can handle much more than 5 MB/s
** source: Single thread SHA256 performance is 99 MB/s using minio/sha256-simd (73 MB/s using crypto/sha256)
** destination: Single thread SHA256 performance is 76 MB/s using minio/sha256-simd (64 MB/s using crypto/sha256)
disk bandwidth - not related, the worst this RAID-5 can handle is over 100 MB/s
LAN issues - on the same LAN and SMB transfers have an average of 10x faster (on big files)
network latency - < 1ms between machines
network load - around nothing, except syncthing
STTRACE=model/all - nothing relevant
GUI open / closed in a browser - absolutely no difference (stuck around 5MB/s)
syncthing version - both are up to date (0.14.20)
At this point, I’m copying the files manually. Even with scanning, it will be 10x faster than letting syncthing do the job by itself. Except the fact I like the concept syncthing (and I use it for other cases, but with few files), it is quite unusable for syncing a multimedia library.
I suspect Syncthing will always perform poorly on network storage as the overhead and latency for the small writes that we do will be the bottleneck, I suspect thats where your IO wait is comming from and is potentially the cause.
If I read his text correctly, he installed Syncthing on the NAS itself. So not really network storage from Syncthings point of view. And his NASes don’t seem to have calculator CPUs either (AMD64).
It’s not unreasonable, probably to large extent due to the block randomization.
Syncthing doesn’t do sequential writes, they are all random writes. RAID-5 is especially bad at this. In RAID-5, each write becomes a data read, a parity read, a data write and a parity write. There is also some CPU overheard, and although the NAS CPU’s aren’t “calculator grade” they are still fairly slow by computer standards - “SHA256 performance is 76 MB/s”.
4.6 MB/s is about 4.6 * 1024 / 128 = 36.8 blocks/s. That then expands into about 150 IOPS due to RAID-5. If those disks are slow spinning drives like WD Green (common in NASes), expect something like 50 IOPS per drive.
If that’s a three drive RAID-5 it’s maxed out at 3*50 = 150 IOPS and we are exactly where we expect. This is with the generous assumption that the fs and RAID layer can actually handle each 128 KiB block as one operation, which may absolutely not be the case. There is also overhead due to fs metadata and so on. All in all, I’m not surprised. Spinning disks suck, RAID-5 sucks, the combination is quite horrible for random writes unfortunately.
Measuring random write performance on the NAS, and checking iostat on it while sync is happening, would confirm.
Thanks for this point of view, even if I’m not sure to understand it correctly.
FYI, the destination NAS is a Synology 1513+ with a RAID-5 (SHR) of 5 WD RED 4TB. BTW, it is formatted in BTRFS.
I’ve made to kind of tests with iostat, giving 3 results each:
global stat (the first one returned by iostat)
syncthing only
syncthing + file copy over SMB
But I’m not sure to understand what IOPS we are talking about, Disks are shown with a maximum of ~45 w/s, while md2 show ~900 w/s. 5*45 = 225, so why 900 on md2 ? Is it a cache effect ?
The most important point in those is the last column of the last two tables - “%util”. It indicates how much of the time the disk (or array) in question is busy servicing a request. For the individual disks in the syncthing only case it’s about 45%. That’s ok, the disks seem to have more to give. But checking the line for md2 it’s at 99% - the RAID array is maxed out. I’m guessing the “lost” performance between that is in waiting. Given that RAID5 needs to both read and write both old data and parity for each write, there will be some time where we’ve done one disk but not yet the the other.
The “IOPS” I’m talking about - I/O operations per second - is what you see in the r/s and w/s columns. The disks are each servicing about 58 + 6 requests, which is consistent with what we expect they should be able to do, being a little bit better than my WD Green guess.
Why so much reading and comparatively little writing? Not sure, maybe this is in the copy phase of syncing?
Regarding the reads, I cannot tell. I cannot install atop on the NAS, which could give me better details. But I’ve tried with htop and sort by disk reads. The only read consumer is syncthing, with an average of 2~10 MB/s.
I did connect to the GUI (climed to 900r/s - 50 MB/s), and the status is clearly “syncing” (only one share, no scan running). After closing the GUI, syncthing still consumes the value above (2~10 MB/s - ~150 r/s).
