I have two Windows systems setup with ST 0.11.3 and they both are setup to sync one directory.
On the first Windows system I placed a 300 GB file. The system is equipped with 24 GB RAM and an Intel Core i7-4790 CPU.
The (I guess) initial scan uses very few RAM and causes a plus between 15-20% on the CPU load. The source file is being read with about 190 MB/s speed. So far, all is good.
Now, once the initial scan is over, syncthing.exe reads and transfers the file with only 1MB/s. A direct file copy via the LAN (Gbit) is maxing out either SDD/HDD or LAN speed, using the same two windows systems.
Also, at the same time the slow transfer starts, syncthing.exe maxes out the i7 CPU and takes around 9-10GB RAM.
What is going wrong here? Obviously a transfer speed of 1 MB/s is too slow and the high resource use is too high for the slow speed.
I know this is Windows, but can you do some command line fu?
C:\> SET GODEBUG=gctrace=1
C:\> syncthing -verbose 1> syncthing.log 2>&1
Post syncthing.log somewhere.
I’m guessing something is badly optimized and churning memory… We keep the full block list for the file being transferred in RAM, which in this case would be about 30010241024/128*40 =~ 94 MiB (one block structure per 128 KiB of file, about 40 bytes per struct) maybe times two for overhead. But still some ways away from 9 GiB.
If that goes fine, the following grabs even more information (but will cause the sync to go very slowly indeed, because the profiling we enable here takes time):
C:\> SET STHEAPPROFILE=1
C:\> syncthing -verbose > syncthing.log
Let it run until memory usage reaches annoying levels. Post heap-<some-number>.pprof which will be created and syncthing.log.
Setup two new Windows 7 64Bit Systems with the latest 0.11.3 and a single share.
Exactly the same result. During the initial scan of the 300 GB test file, all is fine, small CPU % increase and small RAM footprint. Once the scan is complete and the transfer starts, CPU goes 100% and RAM usage is very high.
Figured out the fail. Using SyncTrayzor this comes up with a blanc Syncthing missing all config. Redid with plain Syncthing an produced the log. Link to the log via PM.
New test with a real system setup and a lab setup using the latest 0.11.6 version of Syncthing. Same result as above. Very high CPU and RAM values and slow LAN performance (like 1% of LAN speed).
Probably someone, but not many. According to https://data.syncthing.net, 95% of the population have less than 264 GB synced per device, spread over 135k files. So this is definitely an outlier. I got the log from you, but it didn’t contain any GC trace information and no heap profile…
You don’t say whether the sync that’s ongoing now is the initial one or a subsequent one. I’m guessing not the initial, since it’s been a while. In that case we don’t really expect to max out the network - what happens is that the destination copies blocks that it already has, while the differing ones are transferred. But even just copying and hashing 300 GB on the destination is quite a lot of work, when the file updates. While this happens, not a lot is sent over the network, since there is nothing to send.
You may want something closer to rsync --inplace rather than syncthing, as syncthing always operates on a copy of the file.
I did some more testing. Creating new files and testing with one file each, at 100MB, 1GB, 10GB, 50GB I see a noticable increase in CPU and RAM values. Already at 50GB file size the CPU is maxed out by Syncthing. Also, the bigger the file, the slower the LAN transfer speed, there is definetly something wrong here.
The testing was done on two idle systems, one single share, one file at a time. This should be easy to reproduce. How can I help?
Yeah, confirmed. The benchmarking so far mostly measured the receiving side (since that’s where we can tell when we’re done etc) while the resource problem here is on the sending side. Wonder why, will look into it.
OK, I know what the problem is, there’s a pretty serious performance problem on the sending side for large files. I’ve got a preliminary fix, you can try the build in http://build.syncthing.net/job/syncthing-pr/594/ to try it out. There’s still a memory hit for large files that we should work on, but as long as that “fits” on the system, the actual sync performance should be OK for large files now I think…
I came across this post while looking for slow binary transfers. Had this issue been resolved and fixed in the code?
I am syncing binary file min 100mb+ and my transfer speeds are so slow and it happens in burst but still very very slow. It should not take half an hour+ to send a 100mb file on my network ;(
Sending syncthing(still sending) does not use any cpu (less than 1 percent) and memory usage is max couple hundred meg as far as I can tell. I have 32gb on the sending side and 12gb on the receiving side.
Btw I my system drive is ssd and the binary is on an ssd.
