Scanning of folder is very slow on Windows: about 150KB/s (yes, KB/s)

I’m trying to sync folder which contains 3TB and 120K files (not in flat folder, with 2-3 level hierarhy) on Windows 11 on rather powerful laptop.

Files reside on external HDD, but it could read about 90MB/s without problems and syncthing reports

Hashing performance is 1403.66 MB/s

But still it needs 7 hours for full scan! It is 3’157’285’138 bytes divided by 25200 seconds is ~125289B/s. Or 125KB/s.

Why is it so slow and is it possible to speed it up?

Where is Syncthing’s database located? If it’s also on the HDD, that may slow down scanning significantly, as it’s frequently updated with the scanning / hashing results.

Looks like it is in ~/Local Settings/Syncthing/index-v0.14.0.db (on internal laptops’s drive), as folder in question contains only .stfolder/syncthing-folder-0d7d39.txt and no database.

Please keep in mind that this speed corresponds to reading a single, large file, and not 120,000 of small files. Can you test what kind of performance you get when trying to sync a single, large file?

It is speed when I copy whole folder in question to NUL (Window’s counterpart to UNIX /dev/null), same files which are scanned. They are not very small, as it is music collection and smallest files are 320kbit MP3 songs of order of tens of megabytes, and also there are a LOT of CD images in FLAC, which is hundreds of megabytes each. Of course, there is small files, like playlists, CUE files, etc, but most of the volume consists of 8MB+ files. It is more than 50% of files are 8MB or larger according to my statistics.

I’d still suggest testing with a single, large file first in order to determine the actual maximum speed you can achieve with Syncthing. Also, there is a significant overhead per each synced file, because Syncthing scans, encrypts the data on the source, and then decrypts it on the destination, etc., so it’s expected that the process will be slower than a simple unencrypted local copy.

It is first scan, not synchronization with other node.

I’ll try it with one large file, but I don’t want to cancel current scan, as there is a lot of time invested already :slight_smile:

Honestly, seeing now that the problem is about scanning and not syncing, and judging from my personal experience with external HDDs (e.g. I use two 5 TB external drives with Syncthing on a regular basis), 7 hours to scan 3 TBs of data sounds about right. In addition to that, Windows Defender (or other AV software) also slows the scan down even further.