New test with a real system setup and a lab setup using the latest 0.11.6 version of Syncthing. Same result as above. Very high CPU and RAM values and slow LAN performance (like 1% of LAN speed).
Is there anyone else syncing files of that size?
Probably someone, but not many. According to https://data.syncthing.net, 95% of the population have less than 264 GB synced per device, spread over 135k files. So this is definitely an outlier. I got the log from you, but it didn’t contain any GC trace information and no heap profile…
You don’t say whether the sync that’s ongoing now is the initial one or a subsequent one. I’m guessing not the initial, since it’s been a while. In that case we don’t really expect to max out the network - what happens is that the destination copies blocks that it already has, while the differing ones are transferred. But even just copying and hashing 300 GB on the destination is quite a lot of work, when the file updates. While this happens, not a lot is sent over the network, since there is nothing to send.
You may want something closer to rsync --inplace rather than syncthing, as syncthing always operates on a copy of the file.
I did some more testing. Creating new files and testing with one file each, at 100MB, 1GB, 10GB, 50GB I see a noticable increase in CPU and RAM values. Already at 50GB file size the CPU is maxed out by Syncthing. Also, the bigger the file, the slower the LAN transfer speed, there is definetly something wrong here.
The testing was done on two idle systems, one single share, one file at a time. This should be easy to reproduce. How can I help?
Are you talking about initial syncs here? Random data in the files?
Initial scan works perfect. CPU about 25% and low RAM footprint, exactly with values as you calculated above.
Once the scan is done and the transfer starts, that is what I mean where the problem starts.
To give everyone an idea: Copy of a file over the LAN to the target maxes out my Gbit LAN at about 110MB/s. So no issues there.
Syncthing values, putting a random generated file into the share, conditions as in my last post:
100MB file transfer is instant. 1GB file transfer also nearly instant. 5GB file, CPU 30%, RAM 100MB, transferspeed 30MB/s 10GB file, CPU 30%, RAM 164MB, transferspeed 15MB/s 20GB file, CPU 92%, RAM 590MB, transferspeed 10MB/s 30GB file, CPU 92%, RAM 950MB, transferspeed 10MB/s
Values CPU and RAM from Task-Manager looking at Syncthing.exe
This can be reproduced.
I’ll set up a VM to do some testing, don’t have local disk space enough for it to be relevant. 
Yeah, confirmed. The benchmarking so far mostly measured the receiving side (since that’s where we can tell when we’re done etc) while the resource problem here is on the sending side. Wonder why, will look into it.
1 Like
OK, I know what the problem is, there’s a pretty serious performance problem on the sending side for large files. I’ve got a preliminary fix, you can try the build in http://build.syncthing.net/job/syncthing-pr/594/ to try it out. There’s still a memory hit for large files that we should work on, but as long as that “fits” on the system, the actual sync performance should be OK for large files now I think…
3 Likes
You are a star!
Testing with a 30GB file, not only is Syncthing.exe using 10% CPU, also the transfer speed got much faster with 65-70MB/s.
This fix solved two problems and makes the transfer a lot faster.
Thanks for looking into this, finding the problem and providing the fix in this short amount of time.
2 Likes
Hi
I came across this post while looking for slow binary transfers. Had this issue been resolved and fixed in the code?
I am syncing binary file min 100mb+ and my transfer speeds are so slow and it happens in burst but still very very slow. It should not take half an hour+ to send a 100mb file on my network ;(
I am using v0.14.19, Linux (64 bit) + Win7 x64
Check CPU/memory usage, and network throughput.
Sending syncthing(still sending) does not use any cpu (less than 1 percent) and memory usage is max couple hundred meg as far as I can tell. I have 32gb on the sending side and 12gb on the receiving side.
Btw I my system drive is ssd and the binary is on an ssd.
I have this issue with other devices as well.
Well CPU/memory usage on both sides matters, and generic network throughput matters too.
Understandable however my network speed is not less than 10mb/s and these devices are connected via multiple interfaces (one lan, one wifi)
One device is a quad core +12gb the other is 6cores+32gb
here is one perf on one int
[ ID] Interval Transfer Bandwidth [ 12] 0.0-10.1 sec 1.75 MBytes 1.46 Mbits/sec [ 4] 0.0-10.1 sec 2.38 MBytes 1.98 Mbits/sec [ 8] 0.0-10.3 sec 2.00 MBytes 1.63 Mbits/sec [ 7] 0.0-10.3 sec 2.38 MBytes 1.93 Mbits/sec [ 5] 0.0-10.4 sec 2.50 MBytes 2.03 Mbits/sec [ 10] 0.0-10.5 sec 1.75 MBytes 1.39 Mbits/sec [ 11] 0.0-10.7 sec 2.12 MBytes 1.67 Mbits/sec [ 6] 0.0-10.7 sec 2.50 MBytes 1.96 Mbits/sec [ 9] 0.0-10.7 sec 2.12 MBytes 1.66 Mbits/sec [ 3] 0.0-11.4 sec 2.62 MBytes 1.94 Mbits/sec [SUM] 0.0-11.4 sec 22.1 MBytes 16.3 Mbits/sec
Wifi [ 6] 0.0-10.1 sec 1.38 MBytes 1.14 Mbits/sec [ 3] 0.0-10.1 sec 1.38 MBytes 1.14 Mbits/sec [ 7] 0.0-10.1 sec 1.38 MBytes 1.14 Mbits/sec [ 4] 0.0-10.2 sec 1.38 MBytes 1.13 Mbits/sec [ 9] 0.0-10.2 sec 1.38 MBytes 1.13 Mbits/sec [ 8] 0.0-10.2 sec 1.38 MBytes 1.13 Mbits/sec [ 11] 0.0-10.2 sec 1.38 MBytes 1.13 Mbits/sec [ 10] 0.0-10.2 sec 1.38 MBytes 1.13 Mbits/sec [ 5] 0.0-10.2 sec 1.38 MBytes 1.13 Mbits/sec [ 12] 0.0-10.3 sec 1.38 MBytes 1.12 Mbits/sec [SUM] 0.0-10.3 sec 13.8 MBytes 11.2 Mbits/sec
Well syncthing is not able to utilize multiple interfaces. What about CPU usage on the other side.
Sending side does not use any cpu while sending and I am not seeing much read rate at all
The receiving side is using 2 cores at %100 (but that device holds over 10 shares)
And I am not seeing much read write on the receiving side according to iotop. The file looks frozen at 144mb, it has been there for over 10 mins
Also no errors are reported in the Gui on both sides but the file is frozen pretty much not moving.
It pre-allocates the whole file on startup, so the size shouldn’t change. I guess check the logs on both sides, and provide screenshots from both sides.
Ok this is really curious.
After seeing the file sitting there for like half an hour, I decide to turn the master on the sending side (which I was lucky for because this machine was the one producing this folder most of the time). After that change the file in question was sent fine to the other side. Now why is that? It somehow felt like both sides were frozen and was not able to do much of a progress until I turn the send only.
I am in a hurry and I need to get the file to the otherside so I do not have much time to debug the issue atm, however I hope that my current solution might give you some insight into what the issue might have been.
One more curious situation with another Win10 x64.
The same exact file is stuck on this pc (after it was synced to the Linux pc fine, which was stuck before as well).
Syncthing is showing something like 320mb/s total I/O (in Process hacker and the device has a 512gb ssd) which is insane, and it is using cpu of %25 constantly and the file is stuck at 33.3MB out of 241MB.
I also wonder if Syncthing will shorten the life of my SSD drive.
update: After minutes of constant cpu it moved 3 mb further.