Increase pullers with new version

Continuing the discussion from Slow transfer speed, even on dedicated server:

How can we do the same in the new version?

These options no longer exist.

From The list of syncing files it seems there are only two pullers. I have alot of small files and it is a very slow transfer even though the 2 peers have a direct LAN connection and are very powerful servers.

You now want PullerMaxPendingKiB which is the max number of kibibytes in flight. Multiply by about 16 megs per old style puller thread to get a new starting point. The new default is 32 megs, iirc.

I assume i need to do this on the receiving side?
Or maybe both?

I have set it to 163840 (16MB * 10) on the receiving side, but nothing seems to have changed. Still very slow.

Filesizes are like so:

  • 64.0 KiB
  • 30.0 KiB
  • 6 B
  • 3.41 KiB
  • 5 B
  • 36.8 MiB
  • 20 B
  • 6 B
  • 3.17 KiB
  • 23 B
  • 31.0 KiB
  • 20 B
  • 173 KiB
  • 4 B
  • 6 B

If you have a total of fifteen files comprising only a handful of bytes no puller setting will have any measurable effect. I also have a hard time imagining the sync would be slow due to this.

I have 32994 files with a total of 10.4 GiB. The given sizes were just examples.

Syncthing is running for 2h 40m.

And completed 17% = 15736 files and 2.28 GiB so far.

The download speed shown in the UI is varying between 10 and 100 KiB/s.

The connection speed limit is set to 10MB/s and it is a G-BIT LAN with a fast RAID on both sides.

CPU utilisation is about 1-3% on both sides.

That’s a lot of small files, so you’re probably limited by disk sync writes (database and syncing each completed file). As mentioned in the previous thread, try increasing copiers to handle more files in parallel - more blocks in parallel doesn’t help when the average file just consists of a single block or two.

But if those are spinning disks, the relevant limits are synchronous write operations per second, and each of those are really slow. Network bandwidth etc won’t even factor into it.

1 Like

OIC Copiers still exist.

What is the default value and what value do you suggest?

Only on the receiver or on both sides?

Default is two, I’d start with something like the number of disks in your array. Yeah, this is all on the receiving (“pulling”) side.

But if the RAID is a “stupid” RAID (i.e., hardware RAID controller or NAS as opposed to something like ZFS or NetApp) our disk flushes will probably result in the raid controller flushing the cache on/to all disks, which will hurt performance like hell. So, many small files => slow sync, probably.

Thanks a lot. Speed is picking up now.

I increased copiers in 3 steps to finally 50.

The small files still do little to the overall progress percentage, but i can see that now many files are beeing processed in parallell. This should speed things up.

To see some more progress i also changed download order from random to largest first. :wink:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.