Sync over LAN is very slow

Support
1

Hey all, does anyone have some tips for accelerating syncthing over lan? Right now I’m getting ~60Mbps at peak but I can get ~400Mbps with SMB. I have already turned off relays, disabled compression and compiled the source code and I am using the new “receive only” mode. Neither of my devices is mobile and here’s the benchmark for the slower one (receiver):

type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128 cbc 73405.05k 80972.78k 82906.16k 173196.26k 174200.64k

The receiver also uses a LVM of RAID 0.

2

Usually something is the bottleneck, so start by checking if you’re maxing out CPU, disk I/O on one of the sides. Syncthing is not SMB, neither in how it transfers things over the network nor in how it uses disk I/O.

3

Yes I understand very much that they are not the same, but I expect the performance to be something closer than this (~200Mbps when transferring large files would be good). Looking at htop, iotop and iftop, I think the bottleneck might be IO since Syncthing seems to be constantly read/write different files (probably the ldb files) in multiple threads. And my HDDs aren’t very good at random accessing files. I have turned off rescan for all directories. Is there anything else I can do?

4

Probably not, unfortunately. Syncthing does have a very random I/O pattern which is the opposite of for example SMB. This is, in general, better for large clusters and worse for initially transferring a single thing between two points - and can be painful on HDDs.

If your files are large, enabling large blocks support might help a little.

5

Thanks for the prompt reply! I’ll enable large blocks and see how that goes.

Is it possible to cache the changes to database in syncthing to migrate this issue?

6

Yes and no. Reads and writes are cached as best the database layer can, but with periodical disk flushes. Synced files are also forced to be flushed to disk before we commit the change to the database. Synchronous writes are slow on spinning disks and forces things to wait for them to complete.

7

Is it possible to tweak the frequency of flushing database writes? It seems to me that on my side it’s constantly flushing every second.

8

No; it’ll flush every two seconds or one thousand updates. Are you sure that’s the thing though, and not just normal disk writes to files being synced?

9

Iotop shows utilisation of drives usually. Are you actually running at 100% utilisation?

10

It’s probably every one thousand updates. Right now I have 575,684 files and 93,270 directories (~410 GiB), and ~/.config/syncthing is >1 GiB in size lol.

To be honest I’m not sure. iotop shows that syncthing is using >250% of IO (I have 3 HDDs as a RAID 0 as I’ve mentioned). sysdig -c topfiles_bytes shows 6 different ldb files are being written (I suppose?) simultaneously at 30-40KB/s each (sometimes up to 200KB/s) every second. Is there any other tools I can use to check out my IO usage?

11

I think your bottleneck is fsync() of each file and directory after they are synced, not database writes.

12

Or the overhead of the database for small files.

Iostat shows utilisation

13

I don’t understand how that would explain the low transfer rate when transferring large files. Anyway here’s iostat when it’s transferring a large file at only 3.5MiB/s.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.01   47.91    5.60   22.99    0.00   22.50

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              27.06       131.30       170.93   11738449   15281304
sdb              28.40       299.34        52.40   26760800    4684616
sdc              81.54      4306.43      2754.89  384989960  246284360
dm-0            124.63      4614.95      2810.35  412571177  251241540
14

Iowait of 20% is quite high. You can run with -X 1 option which if memory serves me right gives you periodic dumps. It could be that block shuffling is causing a lot of seeking that slows down disk access.

15

Do you mean syncthing -X 1? That didn’t work. :confused:

16

Audrius means for iostat. Database writes also do not explain slow transfer of large files - they only happen when a file is completed. Blocks are however processed in random order, giving you a seek penalty on spinning disks.

17

Oh nice. It’s -x. Here’s a capture while it’s transmitting a large file in ~60Mbps.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.51   60.80   13.07   24.62    0.00    0.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              6.00    2.00     24.00      0.00     0.00     0.00   0.00   0.00   12.67    0.00   0.08     4.00     0.00   9.50   7.60
sdb              8.00    6.00     48.00    112.00     0.00    24.00   0.00  80.00    3.00    7.33   0.07     6.00    18.67   4.57   6.40
sdc            116.00    6.00   1344.00   2768.00     0.00     0.00   0.00   0.00   20.45   27.33   2.52    11.59   461.33   7.97  97.20
dm-0           123.00   30.00   1272.00   2880.00     0.00     0.00   0.00   0.00   19.71    4.27   2.52    10.34    96.00   6.38  97.60

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.07   64.77    9.84   23.32    0.00    0.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              5.00    4.00     24.00     32.00     0.00     6.00   0.00  60.00   13.60    9.00   0.10     4.80     8.00   9.33   8.40
sdb              9.00    6.00     56.00     84.00     0.00    17.00   0.00  73.91    3.56    7.33   0.08     6.22    14.00   5.07   7.60
sdc            108.00    7.00   1304.00   3212.00     0.00     0.00   0.00   0.00   21.33   29.14   2.54    12.07   458.86   8.63  99.20
dm-0           119.00   23.00   1492.00   3296.00     0.00     0.00   0.00   0.00   19.63    6.26   2.53    12.54   143.30   6.99  99.20

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.51   62.81    9.55   26.13    0.00    0.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              1.00    2.00      4.00      0.00     0.00     0.00   0.00   0.00   28.00    0.00   0.03     4.00     0.00   9.33   2.80
sdb             10.00    5.00     60.00    112.00     0.00    25.00   0.00  83.33    1.20    3.20   0.03     6.00    22.40   1.33   2.00
sdc            106.00   11.00   1364.00   4636.00     2.00     0.00   1.85   0.00   22.00   25.45   2.63    12.87   421.45   8.44  98.80
dm-0           118.00   33.00   1424.00   4752.00     0.00     0.00   0.00   0.00   20.20    4.73   2.56    12.07   144.00   6.54  98.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.52   60.61   10.10   27.27    0.00    0.51

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              4.00    4.00     16.00      4.00     0.00     0.00   0.00   0.00    9.00    0.00   0.04     4.00     1.00   4.50   3.60
sdb             11.00   10.00     60.00    124.00     0.00    24.00   0.00  70.59    1.09    4.80   0.06     5.45    12.40   2.86   6.00
sdc            101.00    8.00   1316.00   1896.00     0.00     0.00   0.00   0.00   22.30   43.00   2.56    13.03   237.00   9.03  98.40
dm-0           111.00   32.00   1244.00   2016.00     0.00     0.00   0.00   0.00   20.43    8.88   2.51    11.21    63.00   6.91  98.80

This does give much more information.

18

Yeah. So sdc (dm-0) is maxed out doing random reads/writes is my interpretation.

19

Yes. sdc is much larger and I only added this drive 2 days ago. So I expect both sda and sdb to be virtually full.

20

Mmm. Unfortunately Syncthing is big on random I/O, which is not friendly (from a performance PoV) to spinning disks.