That explains my 271 MB/s.
I doubt that Syncthing has to wait for my SSD, as it has 2000 MB/s read speed 
and very low access time (NVMe SSD).
Is there a way to disable the weakhash? I set weakHashThresholdPct to 101 but is is still used on scan.
Raise an issue, we should add a way to do that.
1 Like
1 Like
There’s currently no way to disable the weak hashing on the hashing side. The control that is there controls the use of it on the syncing side.
Indeed, and I have that same situation. But I think what we may end up waiting for is the IO system calls and so on, which in Go can mean a bit more context switching and stuff than usual. One of those per 128 KiB block, during which time no hashing happens, is probably the majority of the remaining loss.
I had a couple of branches a long time ago that did this concurrently - one routine to read data and put it in a queue, another one or several that hashed data from the queue. This was not a win at the time due to the overhead of managing that and the locking and communication it needs - the current hashing loop is quite well optimized to run fast and never allocate etc for a single file.
But it does mean we’re limited to a single core for a single file, while running two hashing algorithms. It might be worth it to try again and parallelize it - but I think it’s quite far down on the current list of things that need to happen.
1 Like
The weak hash seems to be somewhat broken.
After the first test yesterday with version 0.14.19 both devices reported 0% sync and 0B/s speed, even after 4 hours (for 36GB in 2 files, Gbit LAN).
Then I set the weakHashThresholdPct to 101 on both devices to disable weak hash for the puller. After that, they did sync and got to “in sync” after a while.
In my munin stats overview from yesterday (for the receiving device), I saw constant 100% CPU usage in the time between the first test and the weakHashThresholdPct setting change, while the disk read throughput was only at 8 - 9 MB/s for the first hour and then 0 until the setting change. After that setting change there was only ~25% CPU load for about an hour. That’s probably the time Syncthing took to sync the VM images. (Also the RAM usage was way higher during that 4 hours)
It would be interesting if you could get the stacktraces as it’s happening or the CPU profile.
@wucke13: Bugs aside, you could set the VM sync folder to a very high scan interval, and use a script in your VM manager that calls the Syncthing API to ask for a rescan once the VM closes. This should help prevent scanning while the VM is being used and make sure a scan is started as soon as it’s necessary.
1 Like
@ProactiveServices That is indeed a solid suggestion, thank you. I will try to modify my KVM script so that it does not modify last changed attribute on the file but to do a touch on shutdown, that will probably cause the same since a scan is only done when the las changed date changes.
I was in the same situation except that my file was only 250mb.
I set the weak hash setting to 101 and it synced fine then. Something is broken there for sure.
I had been battling with this cancerous sync file for over a week, finally it is resolved and I can sleep well.
This could be lib/weakhash: Limit number of hits for any given weakhash by calmh · Pull Request #3925 · syncthing/syncthing · GitHub
That may also explain some CPU usage (gc and map shuffling) but is probably not the whole truth.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.