This is not directly related to the v1.9.0, I just saw it happen again after last release - sorry that I didn’t report earlier.
Yesterday my server (Linux, dual core, 64bit, sufficient RAM) with 12 devices / 20 folders / 2 million (700 GB) files was not responding. Database/system is on SSD, files on HDD. After many tries I managed to get in via SSH to find that Syncthing was the cause. The GUI loaded really slow and every folder had status “scanning”
I restarted Syncthing, paused all devices and scan (I use stacked scanning, so 1 folder at a time) only took 45 minutes, which is typical for my system
The only difference was: the night before I had copied new files that rescan did not pick up. In the morning after the new release the new files were detected and transferred. It seems that the simultaneous updating of database, scanning, transferring, (partial) index exchanging, etc. is too much and each task is dragging each other down.
Testing
startup scan time: 45 minutes
startup scan time with transfers: 3+ hours + heavy i/o
I would be great if Syncthing does not transfer files during (initial) scan. Tthis would help slower systems a lot and I think also faster systems could profit from this.
This sounds to me like just fighting over I/O on HDDs. Reducing the folder concurrency should handle this if the various folders are on the same rotating disks.
Another way is to lower the I/O priority for the Syncthing process.
I have ran it with the lowest CPU/MEM/IO priority in Windows for a few months, and have zero slowdowns even with 20+ folders being scanned and synced at the same time. All of this on a mid-range 4-core system.
@calmh: as I wrote I have set Max Folder Concurrency=1
@tomasz86: I’m running Syncthing with nice/ionice - it works well in daily use. The problem I described happens if all folders are hashed from zero and a transfer starts at same time.
Then I’m not sure what you’re seeing. A single folder can not simultaneously scan and transfer, and if the concurrency is set to one then only one folder can do either of those at the time.
Edit: Oh! Now I see, someone else is downloading from you. Right, yeah, that will add read I/O while it’s scanning. I’m not sure what to do about that. Spinning disks are not great at concurrent I/O.
Just make sure that Syncthing is actually running with the priority that you have set it to. There is a bug, at least in Windows, where the custom-set priority gets overwritten by Syncthing on start unless you disable the setLowPriority option in the Advanced settings. See https://github.com/syncthing/syncthing/issues/6597.
That shouldn’t be possible though: When you have maxFolderConcurrency set to 1, all folders except one should be waiting to scan. If you can reproduce that, please check the config on disk and then provide logs of the startup process until all folders are scanning with model debug logging enabled.