Hello,
I have a very large folder (~70GB) with many smallish files, from around 1MB up through 30MB.
On my Linux box both initial scan and regular scanning completes very fast (at least relative to folder size), less than 20 minutes from start to finish.
On MacOS, however, the sync stalls at 0%; during the hashing process.
So here are the things I have tried to do in getting this folder to sync:
- First I tried syncing to an empty folder on the MacOS machine. This worked fine because there was nothing to hash on the MacOS machine; but transfer was very slow, less than 50,000 files a day (there are around 2,000,000 in total).
- Given this would be complete sometime next month, I decided instead to try rsyncing across the directory, and then syncing.
- This is where we run into the current issue: hashing on the linux box is fast (compared to expectations) - around 13 minutes for total scan; hashing on the MacOS box stalls at 0%. I also had it always convinced that the files on MacOS were āout of sync,ā despite being the same files rsynced across. Note that rsync managed an average of 25MB/s transfer rate; this whole transfer took about 40 minutes.
In trying to fix this slow hashing, I have:
- Tried disabling permissions etc.
- Tried deleting the database on both devices.
- Tried setting the Mac box to send only, and the Linux box to receive only.
- Made sure the Low Priority checkbox was off.
- Tried disabling the MacOS automatic setting of low background process priority with
sudo sysctl debug.lowpri_throttle_enabled=0. - Tried disabling Spotlight indexing, as this was using a lot of CPU and I was wondering if it would interfere with file access times. I assume it was indexing all the new files, no telling how long that would take. Turning it off didnāt reduce the processā CPU usage, though.
The hardware of both machines is new; CPU usage is higher on the Mac than on the Linux box, but this is expected as it is less beefy (it hangs around 2-10% on the Mac, 0.5-2.5% on the Linux box).
Here is a sample of debug output for the scanner - I am not sure what else I can show here:
2022-04-08 11:13:46 walker/ng6wn-5vg5q@0x1400147c000 to hash: User 1/collection.media/0x00122e40.wav.ogg File{Name:"User 1/collection.media/0x00122e40.wav.ogg", Sequence:0, Permissions:0700, ModTime:2021-12-08 09:49:02.843752233 +0000 GMT, Version:{[{OFN3KP7 1649412826}]}, VersionHash:, Length:50527, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:131072, Blocks:[], BlocksHash:}
2022-04-08 11:13:46 walker/ng6wn-5vg5q@0x1400147c000 to hash: User 1/collection.media/A_JojoS01_E05_1_0.06.37.980.jpg File{Name:"User 1/collection.media/A_JojoS01_E05_1_0.06.37.980.jpg", Sequence:0, Permissions:0700, ModTime:2021-12-08 09:56:08.993738134 +0000 GMT, Version:{[{OFN3KP7 1649412826}]}, VersionHash:, Length:15979, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:131072, Blocks:[], BlocksHash:}
2022-04-08 11:13:46 walker/ng6wn-5vg5q@0x1400147c000 to hash: User 1/collection.media/A_Assassination_classroom_E10_1_0.19.47.110.jpg File{Name:"User 1/collection.media/A_Assassination_classroom_E10_1_0.19.47.110.jpg", Sequence:0, Permissions:0700, ModTime:2021-12-08 09:50:42.887082262 +0000 GMT, Version:{[{OFN3KP7 1649412826}]}, VersionHash:, Length:18146, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:131072, Blocks:[], BlocksHash:}
2022-04-08 11:13:46 walker/ng6wn-5vg5q@0x1400147c000 to hash: User 1/collection.media/Dorohedoro_01_0.20.35.734-0.20.39.134.mp3 File{Name:"User 1/collection.media/Dorohedoro_01_0.20.35.734-0.20.39.134.mp3", Sequence:0, Permissions:0700, ModTime:2021-12-08 10:04:06.933722393 +0000 GMT, Version:{[{OFN3KP7 1649412826}]}, VersionHash:, Length:56509, Deleted:false, Invalid:false, LocalFlags:0x0, NoPermissions:false, BlockSize:131072, Blocks:[], BlocksHash:}
Output of time ls -lR on the directory on the linux bos:
real 0m13.249s
user 0m8.612s
sys 0m4.075s
On the MacOS box:
real 5m18.654s
user 0m22.367s
sys 0m54.529s
Even accounting for the processing gap, a difference of ~13 minutes for initial hashing vs. 0% after sitting there for ~12 hours seems off to me! If the ls output is anything to go by, I would maybe exepect around 5-10 hours for initial hashing. Would I be right in thinking that this process is single threaded, as well? If that is the case, then the two processors are almost identical in SC performance on basically any benchmark I have thrown at them.
I am suspecting some weirdness with MacOS and file access at present.
Please let me know what other info I can provide/things I can try. I am aware this might be a somewhat absurd case; in the future this folder will be a lot smaller, but I want to be able to be doing the things necessary for that on both devices 
