Extremely slow GUI Windows

Support
1

Hello,

We are using syncthing in a rather unconventional way, where we have one “central” hub and about 100 other devices. No synchronisation is going on between the devices except from “central hub” to device and vice versa. In total we have around 1000 folders synchronising, on average about 6 folder per device with some extra folders for some devices.

Everything is running on Windows Server 2016.

What we can see is that when we open the GUI on the central hub, it takes around 10-15 minutes where all folders are in the state “Unknown” and maybe another 10-15 minutes (on a normal day) before all folders have their “correct” status.

We can not see anything in terms of CPU Usage, RAM Usage or Disk Usage that should be an issue, CPU is usually around 20%, RAM about 20% used by Synchting, total 50% RAM usage and average response time on Disks (for syncthing.exe) are 1-5ms.

The other unconventional part about our setup is that the average response time to the devices are around 900ms, as the devices are remote without normal connection possibilities.

Lastly, all the files that we are synchronising are on an Azure storage so the access time that way are a bit slower than having the files on the local machine.

On the central hub we are using Syncthing 1.10.0 and on the remote devices it is currently using 1.3.4 (will be upgraded rather soon).

Do you have any ideas as to how I can increase the performance on the central hub? We are currently using Watch for Changes on all folders, which may be the culprit but it is rather important that files are transferred as soon as possible instead of on a schedule.

I would gladly get some logs, but I don’t fully understand all the flags available and since the GUI is so incredibly slow it takes very long time just to get the settings or advanced settings popup to show.

If there is anything else needed from me, please tell me.

Thanks in advance

Edit: One more thing, when I tried to do the API-call to /rest/db/status with a folder as parameter, I left it running for 5 minutes without getting a response whatsoever.

2

The web UI not scaling well is a known problem, nothing to do there. There’s an alternative web UI that is aimed at large deployments that only shows status (no config modification): https://github.com/kastelo/syncthing-tech-ui From what you describe there’s no reason to assume there’s any problem with the performance of Syncthing itself, it’s just the GUI. Except maybe for

Does this happen while the GUI is open? Then it’s not surprising, otherwise it definitely is.

Same here: At the moment there’s no need for logs, the slow web UI is not unexpected. For such a large deployment you should use the API to interact with Syncthing. And/or rethink your setup, involving multiple central nodes to distribute the load, though I’d expect a thousand folders to still be manageable.

That’s highly recommended :wink:

3

Wow, that’s awesome. I’ll try that immediately as that is one thing we have been looking all over for. Apparently my google-skills weren’t good enough for this.

After testing a while it seems like it is also rather slow to get the API-calls through. After waiting for 5 minutes it has so far gotten to 36 folders, and all the devices. The first few times it went rather smoothly to get all the folders but it seems to be having some issues still. The devices are shown after 3-4 seconds but the folder takes a very long time.

I tried once again now for a few folders. It is almost instant if I close all GUI and also close the “Tech-UI” you linked to above. If it is open, it took about 1.5 minutes before getting the response.

I know, but it seems hard to get the gist of how it’s working. Adding Devices, adding folders with the correct ignore patterns and the correct send-receive/sendonly/receiveonly and correct path is probably very simple with API, just that I can’t seem to find the correct endpoints and queries to get it proper. But that’s probably on me :slight_smile:

Thank you very much for your answers and the (extremely) quick response

4

Seems like I can’t edit responses after a certain time, but tested the Tech-UI a bit more and see these issues arising probably showing the reason for the issues I have:

Capture
Capture1903×174 47.7 KB

We can see that the API-call that the UI makes takes 50 seconds per folder in this particular time of the loading of folders, which makes me believe that something is wrong with our setup rather than the UI.

Is there any kind of enhancements that I can change to make it faster?

I have done the following: maxFolderConcurrency: -1 (All folders simultaneously) setLowPriority: False

Could it be something with databaseTuning? As our database is about 10GB. It is set to Auto which probable should use “large” automatically.

StartBrowser is set to true as well, could it be that it opens a “fake” browser with the GUI that slows it down?

I feel a bit lost in all this, sorry for my confusion :slight_smile:

Edit: Last update probably. Increased Logical Cores to 16 and it works as if it was new with only 10 folders again. So the solution to this problem was to increase the Cores, and I’m super happy again :smiling_face_with_three_hearts:

5

Running with unlimited folder concurrency while having many folders will just cause tons of disk thrashing.

6

I changed to 0 and so far it seems to work better, except for the obvious part that very many folders are “Waiting to scan”. Does this mean that they have to scan to be able to sync the files? What I mean is that since we have Watch for changes enabled, will the files still be synced as soon as possible after creation or does it have to scan the folder before “finding” the new files?

Since we have 16 logical cores, it is currently scanning 16 at a time which takes a while to scan 1000 folders.

7

Pulling (downloading) and scanning are mutually exclusive operations. If it’s waiting to scan, it will not download anything.

If you are getting “waiting for scan”, then things will not be immediate, as even with change watching enabled, it needs to scan the file before it’s able to tell others about it.

8

That’s a shame. I understand why, but it makes it hard to get working as we wish. With 16 cores, is there any recommended “max” folders to find a reasonable middle way where the disk thrashing is “low” yet more than 16 folders are scanned at the same time? As we have so many folders which some are quite large as well it takes a while for the system to scan them.

Another question regarding the same: If we disable or set ScanIntervalS to once every day, would that make the “Waiting to scan” appear less often and keep the sync immediate as much as possible?

9

The initial and periodic scan checks the entire file tree. Watching for changes checks only the paths where changes occurred. Thus after the initial startup, I’d expect changes to be picked up quickly without much waiting.

Increasing ScanIntervalS will reduce the amount of full scans done. However even a full scan shouldn’t take that long. What kind of storage are we talking about?

10

The storage is currently about 1.5TiB of files on an Azure server, unfortunately I don’t know the exact specification of said server in terms of performance as it’s not under our control.

11

Ah right, you wrote that before - I forgot sorry. Well less surprising that it is slow then. However I wouldn’t expect it to be faster when increasing the concurrency, just going slower over more folders at a time. Also did you check that filesystem watching even works on azure storage?

12

I would not worry about the CPU that much. I have a weak 2-core CPU processing ~30 folders at once, which would translate to 240 folders in your case, and the bottleneck seems to always be the storage anyway. I have never seen the CPU being pegged during the process.

On the other hand, if you have a very slow HDD, then there may be no difference in the processing speed between a 1-core and a 16-core CPU.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.