Scanning slow on MacOS

Support
1

Hello,

I have a Thunderbolt Raid storage connected to my iMac. The storage is build with WD GOLD drives (12x18TB in a RAID6), it meets all the test requirements of blackmagic speed tests so the read/write is definitely fast. I have 128GB RAM. I installed Syncthing to share a folder with friends, added the folder to Syncthing and got scan time over 1 month. The synchting read/hashing speed is according the UI 56KiB/s. The folder is having ~300GB and 11k files. The CPU is 90% idle, the Syncthing processes are on CPU 0.1% used (idle), the bytes written/read are 60kb/10Mb in ~1min. The storage itself is used by multiple processes but any new process is capable to read with at least 40MB/s. I am using v1.19.0, macOS (64-bit Intel/AMD). Any idea how to improve the scan speed? Thank you!

Boris

2

I don’t know precisely as it seems way slower than expected, but the concurrency settings for MacOS are set quite low by default, while your array should support lots of concurrency. In advanced settings, for the folder in question, set Hashers (number of concurrent hashing threads/routines) to something like 8 or maybe 12 to see what happens.

1 Like
3

Thank you! So by default there is a 0. Setting it to 1 is providing speed of amazing 18KiB/s, 12 hashers provide 26KiB/s, the fastest value I was able to get was with 6-8 hashers (~160KiB/s). At that speed the hashing would complete in ~20 days which is still insane slow. If I take shasum for the hashing on a file, then I am getting over 100MiB/s, if I take openssl sha1 on another file (1GB both files) I am getting over 300MiB/s hashing speed. If I do it with 10 files in parallel, using shasum or openssl sha1, I am getting the overall speed similar to the values I mentioned above. So there is no bottleneck on the CPU or disk speed. I tried to delete the folder from Syncthing, create it again under another name, I tried to restart Syncthing, I tried to set it to Send/Receive folder type (the folder will be shared as Send only), nothing improved the speed.

4

Yeah, I have no idea what’s going on. Sorry.

5

Maybe some of the profiling options could help here? https://docs.syncthing.net/users/profiling.html

1 Like
6

Do you have a lot of tiny files?

Is Syncthing database on the same array?

Wonder if case sensitivity has something todo with this.

Whats the filesystem on the array?

7

Thank you Andre, I will gather the logs.

8

Majority of the files are somewhere between 6MB-120MB but there are of course smaller and bigger files as well. The database is by default on MacHDD (ssd) and I have not changed this default so it is a different place. The filesystem is NON case sensitive APFS and while I am aware about performance limitations when you have a lot of files and directories on HDD and not SSD this seems to be not the case as listing all the files in all the subdirectories or copying them works as expected fast. I can imagine that lot of directories or files would cause the performance issues but we are talking about 11k files in 347 directories.

9

I have the pprof files Andre, where do I send/upload them? Thank you!

10

Good question. Sorry I only read about the possibility to generate profiles, but have never worked with them myself. If you cannot attach them here in the forum, would something like Dropbox with a public link be feasible?

1 Like
11

Attached. Does it help? syncthing-cpu-darwin-amd64-v1.19.0-132738.pprof (2.3 KB) syncthing-cpu-darwin-amd64-v1.19.0-132816.pprof (1.2 KB) syncthing-heap-darwin-amd64-v1.19.0-133028.pprof (48.4 KB) syncthing-heap-darwin-amd64-v1.19.0-133034.pprof (49.1 KB)

12

Can you try toggling this setting via advanced settings: caseSensitiveFS — Syncthing v1 documentation

This does lead to dangers of file loss if the file exists in multiple different cases on a case sensitive system, but just curious to see if this helps in any way.

13

Sorry I can’t see any clues in there, just that it seems to be a really small sample? Hope somebody with better Go knowledge can make sense of the numbers.

14

If you scan a similar sized folder that is on the internal SSD, how’s the performance then?

15

Thank you Audrius, this improved the situation so from ~160KiB/s I am at ~1MiB/s. But I do not plan to use something what is not safe so I am back at the option disabled. I did couple of tests on the HDD storage and on dedicated SSD (usb c), both using case insensitive apfs. 1GB on the volume - unable to measure as it completes almost instantly. 10GB on the volume - ~5MiB/s on HDD storage and ~160MiB/s on SSD. 20GB on the volume - ~200KiB/s on HDD storage and ~150MiB/s on SSD. 300GB on the volume - ~160KiB/s on HDD storage and ~50MiB/s on SSD. I do not have enough disk space to test it on internal SSD but I assume the external SSD provides similar results. All the volumes are excluded for Spotlight and the Spotlight is disabled on my system (so no indexing of the files is happening).

16

I’m not a mac user, but is there something like iotop/iostat to check io wait?

17

There is actually another command which can show the process itself and what happens with the read/write etc.

sudo fs_usage syncthing

which in my case has an output like:

16:32:36.098645  write             F=32   B=0x1000                                                                                                                                                                            0.000008   syncthing.1030685
16:32:36.098658  write             F=32   B=0xc97                                                                                                                                                                             0.000007   syncthing.1030685
16:32:36.098667  read              F=32 [ 35]                                                                                                                                                                                 0.000001   syncthing.1030685
16:32:36.138932    THROTTLED                                                                                                                                                                                                  0.715509   syncthing.1030732
16:32:36.138933  read              F=25   B=0x8000                                                                                                                                                                            0.872152   syncthing.1030732
16:32:36.224242    THROTTLED                                                                                                                                                                                                  0.794747   syncthing.1030714

Any THROTTLED as far as I know means that the MacOS is throttling resource heavy processes for idle applications. Why the Syncthing is considered as an idle application I have no clue, the developers can probably answer and fix this.

How to temporary fix it?

Prior folder scan you can run:

sudo sysctl debug.lowpri_throttle_enabled=0

To enable the throttling again you run:

sudo sysctl debug.lowpri_throttle_enabled=1

You can as well run

sudo sh -c 'echo debug.lowpri_throttle-enabled=0 >> /etc/sysctl.conf'

to disable the throttling on your system completely (no clue what will be the long term impact).

With disabled throttling the scanning performance is in my case ~270MiB/s (the value 8 for Hashers seems to be good for my storage).

Thank you very much everyone for your help!

18

Cool. Now we can also deal with broken powersaving “features” on non-Android platforms, too. Yeah! /s

A more serious take on this issue: what is the priority of the Syncthing process on your system?

19

The priority of the process is the default coming from the package (no idea how to check).

20

Syncthing lowers priority be default; there’s a setting to disable it too (Advanced → Options → Set Low Priority). It doesn’t usually have this drastic of an effect though, so still not sure what’s going on with this system. Perhaps it’s heavily loaded in some way so the OS things it needs to throttle everything nonessential?