380k files, 24GB in directory results in very slow sync = 8 days

Support
26

so, here are some results && initial conclusions, probably some of you can add more.

  1. this machine was CPU constrained during Scanning&Synching, because of (random order):
  • slow caseinsesitive comparison calls
  • long filenames of 380k files
  • slow cpu
  • only one core used
  • multiple os.reaadir calls during Scanning

  1. enabling caseSensitiveFS (which is not default in syncthing), reduced Scanning time from 105 minutes to 3 minutes, and Syncing was more/less very fast.

  2. after it, syncthing is still cpu constrained during Scanning, but time needed to complet Scaning (and later Syncing) is not only acceptable, but I would say fast, in regards to amount of data (380k files with long filenames (sic!) i.e. maildir) being processed.

  3. syncthing still is using only one core (per folder) during Scanning, and my guess is that with caseSensitiveFS=true, now the os.readdir is main culprit. It may be, or may not be, due to the logic further optimized in syncthing, nevertheless: 3min is fast so I would say: not worth it.

  4. some of configuration changes were good in general (setLowPriority=false, databaseTuning=large, Changes, copyRangeMethod), which helped to speed up Scanning from initial 160min to 105min. I’m keeping them.

  5. the biggest single winner of course is caseSensitiveFS=true

  6. I reverted to default disableFsync (being false). While it would provide some speedup (during Syncing) it’s not worth the risk.

  7. many settings I tried had no effect (or to small to notice) on my machine, due to constraint being on CPU, not IO or due to the fact that only 1 core was used or due to the specific makeup of my data (maildir). I reverted them to defaults: disableTempIndexes=false copiers=0 hashers=0 maxConcurrentWrites=2 SendReceive fsWatcherEnabled=true

Attaching pprof file for the caseSensitiveFS=false scenario, here is quick peek for the impatient ones syncthing-cpu-linux-arm64-v1.27.2-180504.pprof (23.7 KB) :slight_smile:

(pprof) top
Showing nodes accounting for 24.80s, 69.66% of 35.60s total
Dropped 262 nodes (cum <= 0.18s)
Showing top 10 nodes out of 73
      flat  flat%   sum%        cum   cum%
     4.74s 13.31% 13.31%      4.74s 13.31%  runtime/internal/syscall.Syscall6
     3.41s  9.58% 22.89%      5.17s 14.52%  strings.(*Builder).WriteRune
     3.29s  9.24% 32.13%      4.29s 12.05%  runtime.mapassign_faststr
     2.95s  8.29% 40.42%      3.46s  9.72%  runtime.findObject
     2.15s  6.04% 46.46%     15.69s 44.07%  github.com/syncthing/syncthing/lib/fs.UnicodeLowercaseNormalized
     2.02s  5.67% 52.13%      2.02s  5.67%  golang.org/x/text/unicode/norm.(*input).skipASCII (inline)
     1.85s  5.20% 57.33%      1.85s  5.20%  unicode.ToLower
     1.79s  5.03% 62.36%      1.79s  5.03%  unicode.ToUpper
     1.30s  3.65% 66.01%     29.61s 83.17%  github.com/syncthing/syncthing/lib/fs.newCaseNode
     1.30s  3.65% 69.66%      8.03s 22.56%  os.(*File).readdir
3 Likes