Massive CPU spikes since 1.27.8 update

Hello, I am experiencing massive CPU spikes since updating to 1.27.8. There are no major changes to my folders or configuration, and all other computers with the same sync folders are operating as usual. I am attaching a 30-day CPU usage graph to show the massive impact of the last update. The exponential increase of CPU usage is when I updated Syncthing sometime late June. What can I do to troubleshoot this?

This issue is persisting after restarts.

Thank you!

Mac mini (Late 2014) running macOS 12.7.5

I’d start by looking at what it’s doing, in the GUI and logs.

Hi Jakob, the GUI is not showing anything out of the ordinary, and every folder appears up to date without any scanning in progress.

Log is frequently showing “Connection from at : (tcp-server) rejected: already connected to this device” around every 20 seconds or so. Is this normal?

I am not familiar with reading logs, but I can try to take a look. What debug data would be helpful for me to start with?

May or may not be normal depending, but certainly won’t cause 320% CPU usage. Running it with -verbose would give some more log data on what it’s doing, all the debug options are too verbose and too technical to begin with.

Potentially some other device experiencing some kind of failure could be stressing it by repeatedly downloading data, though I’m not sure why that would happen either. If that were the case it should be visible in the GUI as well.

I just want to quickly confirm this issue is specific to version 1.27.8. A downgrade to either 1.27.6 or 1.27.7 fixes the issue. It comes back the moment I upgrade to 1.27.8.

Since 1.27.9 for macOS should be around the corner, I will be happy to continue investigating if the problem persists after the update.

Besides the verbose log, a CPU profile could also shed some light: Profiling — Syncthing documentation

1.27.9 has been out for a while. The differences between 1.27.7 and 1.27.8 are very minor. (1.27.8 to 1.27.9 even more minor.)

One thing that could be relevant, pretty much the only one, is that Syncthing now sees and reacts to changes to extended attributes, but if that were causing some issue it would manifest as a lot of scanning which you would see in the GUI.

I have updated to 1.27.10 and the computer is still experiencing the same issue. My only solution so far is to downgrade to 1.27.7.

I’ve captured a CPU profile hoping that someone can help me to shed a light on this.

Thanks in advance!

syncthing-cpu-darwin-amd64-v1.27.10-234636.pprof (9.8 KB) syncthing-cpu-darwin-amd64-v1.27.10-234712.pprof (16.1 KB)

This is one weird profile:

At first I thought it just means syncthing is idle, and the little bit that’s happening is runtime and a little quic-go work. However this accounts for 31s of cpu time in a 30s interval, so definitely not idle. It looks like quic is resetting a timer a lot, and that may also be the cause of the runtime part.

Can you disable quic and see if it still happens? To do so replace “default” as listen address with “tcp:0.0.0.0,dynamic+https://relays.syncthing.net/endpoint”.

1 Like

@imsodin Thanks for looking into this!

Looks like we’ve found the culprit. I have disabled quic and the CPU usage returns to normal. Was there a change since 1.27.8 that might have caused this?

1 Like

@eeiciap you should open an issue and add your profile:

Please wait with that for a sec. on mobile right now, will write why/what else after.

Reason being is that we are not on the latest version of quic-go, but it was updated recently. As in the next syncthing release will have the latest quic-go. @eeiciap could you wait for that, and then see if it happens again. And if it does, please check out verbose logs as suggested above by calmh. Generally it’s a good idea to do a minimum of debugging on our side before reporting to dependencies - we might be doing something problematic and it’s not caused by anything in their lib.

Resp. of course if you depend on quic, you can also downgrade again until the new release is out or already start investigating logs now.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.