Very high CPU usage

We are syncing about 50.000 Files over 30 machines. Ulitimately we have a constant very high CPU load (50%) which makes all very slow and sometimes unusable. Can anyone give me a hint what the problem might be and how to fix that?

Your screenshots shows a folder is scanning - during that time it’s expected that cpu load is high. Are you saying it keeps scanning all the time?

I deleted some conflicting files which caused a constant scanning on the server. That solved the problem for most of the clients. But some of them (including the server) still have a high CPU load even when they are up to date:

Are any of the connected devices still syncing? When they request data, this data is validated by (weak) hashing. I wouldn’t expect that to be so costly, but it’s the only thing I can think of. You can capture a cpu profile as described in the docs and post it here to find out what’s actually causing the load: https://docs.syncthing.net/users/profiling.html

1 Like

You are right, there are some clients which keep syncing!

Here comes the cpu profile …

syncthing-cpu-windows-amd64-v0.14.52-095614.pprof (62.3 KB)

Wow…

Indirectly the syncing clients are the cause, but it’s not hashing, it’s computing their completion that takes so much time. Most time is spent unmarshaling db objects and with leveldb operations themselves. Good news is, that this should only use resources when the web UI is open (not entirely sure though).

@calmh Do you know if iowait contributes to the cpu time in the profile?
Then a major part of that cpu percentage isn’t actual computation time - it’s due to a bottleneck in disk access. Apparently not, see below.

I don’t think iowait is counted neither in the profile nor our own CPU gauge…

Closing the GUI didn’t have an effect on the CPU usage:

grafik

But obviously syncthing is writing a lot to the disk:

FWIW I think there’s a five or so minute grace period before the GUI is no longer considered attached

Is there any way to find out if another machine is still accessing the GUI … just to make sure I’m not waiting in vain :wink:

As it’s writing to db I don’t think you can do anything except wait until the sync finished. If you want to speed things up, the db needs to go onto faster storage (ssd).

I doubt that the sync will finish as the clients are trying to sync only a couple of files which didn’t work the whole weekend. I can’t believe that this is causing the problem. Or am I mistaken!? Maybe another problem?

grafik

Agreed, that sounds weird.

Is the disk activity screenshot above from the same time as the cpu profile? As the profile does not indicate any writing activity.

Also check the other device’s web UI to see what it is actually doing.

Sorry for the german screenshot. There is no reading activity. Lesen - reading Schreiben - writing. Tomorrow I will try to make a screenshot of the server and a client at the same time. The disk activity is from more or less the same time. But I can try to make more screenshots tomorrow.

No worries, I speak German :wink:

I was asking exactly because there is writing activity only, while the profile shows reading only.

Good point! I’ll check that tomorrow … and send you brand new screenshots :slight_smile:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.