Poor performance on big file


#41

That explains my 271 MB/s.

I doubt that Syncthing has to wait for my SSD, as it has 2000 MB/s read speed :slight_smile: and very low access time (NVMe SSD).

Is there a way to disable the weakhash? I set weakHashThresholdPct to 101 but is is still used on scan.


(Audrius Butkevicius) #42

Raise an issue, we should add a way to do that.


#43

(Jakob Borg) #44

There’s currently no way to disable the weak hashing on the hashing side. The control that is there controls the use of it on the syncing side.

Indeed, and I have that same situation. But I think what we may end up waiting for is the IO system calls and so on, which in Go can mean a bit more context switching and stuff than usual. One of those per 128 KiB block, during which time no hashing happens, is probably the majority of the remaining loss.

I had a couple of branches a long time ago that did this concurrently - one routine to read data and put it in a queue, another one or several that hashed data from the queue. This was not a win at the time due to the overhead of managing that and the locking and communication it needs - the current hashing loop is quite well optimized to run fast and never allocate etc for a single file.

But it does mean we’re limited to a single core for a single file, while running two hashing algorithms. It might be worth it to try again and parallelize it - but I think it’s quite far down on the current list of things that need to happen.


#45

The weak hash seems to be somewhat broken. After the first test yesterday with version 0.14.19 both devices reported 0% sync and 0B/s speed, even after 4 hours (for 36GB in 2 files, Gbit LAN). Then I set the weakHashThresholdPct to 101 on both devices to disable weak hash for the puller. After that, they did sync and got to “in sync” after a while.

In my munin stats overview from yesterday (for the receiving device), I saw constant 100% CPU usage in the time between the first test and the weakHashThresholdPct setting change, while the disk read throughput was only at 8 - 9 MB/s for the first hour and then 0 until the setting change. After that setting change there was only ~25% CPU load for about an hour. That’s probably the time Syncthing took to sync the VM images. (Also the RAM usage was way higher during that 4 hours)


(Audrius Butkevicius) #46

It would be interesting if you could get the stacktraces as it’s happening or the CPU profile.


(Adam Piggott) #47

@wucke13: Bugs aside, you could set the VM sync folder to a very high scan interval, and use a script in your VM manager that calls the Syncthing API to ask for a rescan once the VM closes. This should help prevent scanning while the VM is being used and make sure a scan is started as soon as it’s necessary.


#48

@simplypeachy That is indeed a solid suggestion, thank you. I will try to modify my KVM script so that it does not modify last changed attribute on the file but to do a touch on shutdown, that will probably cause the same since a scan is only done when the las changed date changes.


(totoba) #49

I was in the same situation except that my file was only 250mb.

I set the weak hash setting to 101 and it synced fine then. Something is broken there for sure.

I had been battling with this cancerous sync file for over a week, finally it is resolved and I can sleep well.


Out Of Memory -> Panic.Log
(Jakob Borg) #50

This could be https://github.com/syncthing/syncthing/pull/3925

That may also explain some CPU usage (gc and map shuffling) but is probably not the whole truth.


(system) #51

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.