Hello there. I am trying to sync folder that have like 500k files, a lot of small ones, and about 200GB of data, and problem is that every scan of these files takes ages (hours) and a lot of cpu time, while i’ve seen that syncthing uses file data + size for it’s check, so should be a lot faster. It’s nvme ssd (NM790), and copying that folder/compressing etc. is very fast, in minutes. Heck, even winmerge does it’s scan in couple of minutes.
Thing that adds to that problem is that continous scanning of files does scan WHOLE folder on single file change - why? It’s like i touch one file and it goes for hours to rescan again…
Another thing is syncing - it also literally takes ages (again, hourse), while doing smb/sftp etc takes minutes - looks like small files are very cumbersome for syncthing - is there any reason for that?
Newest syncthing on all devices, which are on windows 10/11 without any special setup etc., mix of 1 and 10gbit nics, lan+wan syncing test. HW is on newest am5 platform with 9950x3d, 96gb of ddr5 ram.
If you need any more information - i can provide, just tell me what
Thanks in advance!
During a scan, Syncthing is hashing the files it is scanning. Moving files with SMB or SFTP doesn’t do that.
Please share the specific CPU. Sometimes the “latest” build of Syncthing turns out not to be — please share the specific Syncthing version on this Device.
What is your CPU load like during hashing?
Have you made any changes to Syncthing’s configuration for performance? Take a look at Configuration Tuning — Syncthing documentation . Consider testing performance after setting setLowPriority to false (allows Syncthing to use more CPU) and setting hashers to a larger figure — I’d start with the total number of CPU cores and see how that works.
It likes to eat whole cpu core, so i suppose it uses that cpu cycles. I mean hashing can be slow, but not THAT slow, like 10-20MB/s average, while i can read and hash in other software same folder with usually 1GB/s+ speeds (depends which files it hash in current time). Problem is also with “Watch for Changes” - whenever i touch one file it tries to hash check whole root folder - why? Touching a 1KB file shouldn’t invoke hasing 500000 other files.
i typed my specs in 1st post - 9950x3d. Syncthing version:
v2.1.2, Windows (64-bit Intel/AMD)
“Hafnium Hornet”
Build 2026-06-26
And yes - i understand how it works. I remember i tuned stuff, but that was AFTER i noticed such low performance in any set profile like scheduled scan/watch scan, so it’s not tied to that knobs i modified.
I suggest just putting 1m files with random sizes (from /dev/random i.e.) totaling 200GB and trying to synchronise between 2 pcs - you will see how slow it goes.
I am trying to sync(it had older version of folder) laptop after a while and syncing 480k files and 330GB of data took about 6.5h - i don’t think syncing 20 files/s is fast in modern standards. When files are large, transfers are fine, it’s just files/s is VERY slow. Is there any specific reason to that? I disabled block indexing but it didn’t change anything. @acolomb
Congratulations to hashing speed of 10-20MB/s
My current speed is never more than 4MB/s, usually around 1.5MB/s. I have currently around 1.5 million of files, and it can’t finish the scanning in less than 24 hours, so the next scan will start sooner than the previous has ended. So syncing of my new files has even stopped…
I was thinking unraid + syncthing is fine for some back-up purposes (especially with excluded files functionallity), but that’s not true in my case. But as I started to increase the number of files in the folder, the slower it started to be.
Some changes have landed recently that should improve the performance with large file counts. Stay tuned and have a look at the release notes for the next release. We’re getting better step by step.
I hope so. I know it’s open source project, but still, having a folder to sync in such a long time is little bit of a pain. Will look forward for patches - thanks
Thanks in advance!