Speed up transfers of large files

I searched Google and all I saw were old posts talking about raising “Listeners” and “Copiers” which I am not sure if newer version has this? Last I read was max folder concurrency, but I want to do “copiers” per-file (large 2gb+ file) to raise my transfer speeds. How can I achieve this in version 1.8.0, Linux (64 bit)?

What filesizes and throughput rates are we talking about? I am doubtful that any settings will improve the speed. The only possibly relevant setting is pullerMaxPendingKiB. Increasing it (default is 32MiB) will allow more blocks to be synced in parallel. Though for large blocks I wouldn’t expect block overhead to have any effect. You also need to increase maxRequestKiB to the same value on the device you are syncing from (respectively just increase both on all devices).

I am seeing range of 170~KiB up to 1.80~MiB when I have 400/25 speeds. File sizes are over 2 GB+ (.ts and .mp4)

Let me try your suggestions, will post results.

Thanks.

EDIT:

Raised to 64000 KiB, but I am seeing similar numbers and it still keeps fluctuating. Is that the best I am going to get with 25 MB upload or can I continue to raise to see if I get any improvements? Also, there is a “Listeners” that’s showing 3/3, is there a way to raise that maybe? How exactly do these “Listeners” work?

As I said, I didn’t expect this to make any difference, so I don’t expect raising it even more will. Did you check your actual network throughput? Usually the carriers give speeds in bits (because higher numbers, yeii), so that’s ~4MB/s which is just twice what you see.

And I assume you already have a direct tcp connection, not a relay one.

Correct, direct connection.

So from your experience what is the “sweet spot” to set this? Is it the default, 32MiB? Or will 64MiB help in certain circumstances?

What about Listeners, is that something that can be used to advantage? Maybe have parallel connections from same device if possible?

I’m not worrying about cluttering my bandwidth, syncing is my #1 priority.

BTW, these are unRAID multimedia servers, so large files are dominant.

Listeners has nothing to do with this. And I haven’t done any tests on this, because I don’t see a reason to. The default has been chosen for a reasonable limit for in-flight data and thus memory usage, without impeding syncing. So in my opinion it’s the wrong approach to think about “what should I tweak this and that option to”, you should think about “what is limiting my sync speed”. As a baseline, determine the direct network bandwidth. Then observe the real speed and resource usage. Either that clears things up for you already or you describe that here and we can start thinking about whether/what in Syncthing is limiting it.

1 Like

Also depends on what you’re syncing. If it’s changes to existing files the dominating factor will be disk I/O, not network transfer.

@imsodin You’re right, didn’t think about that… I have a feeling the new dd-wrt fw I installed lowered my speeds, I should do a speedtest since I haven’t checked it lately…

@calmh Mostly new files syncing from main unRAID server to backup unRAID server (these are both running the latest docker 1.8.0)

“dd-wrt” makes me think these may not be the most powerful machines. The crypto and hashing requires a fair amount of cpu.

dd-wrt is the fw on my routers. Machines both have AMD FX8350 so yeah, they are not the greatest, but I do have the advantage of 8 cores…

EDIT:

Sure enough, my speeds lowered. So I took your advice and left it 0 (default 32MiB) and going to reflash previous version of dd-wrt on my routers.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.