Slow scanning speeds on NTFS

[chamber2038]

Firstly, this seems to be an issue with my current Linux setup instead of Syncthing itself. But I’m unsure of how to troubleshoot this. I have searched on the internet for a solution but could not find any. So writing here in hope of finding some answers.

I’m receiving extremely slow scanning speeds from my NTFS drive. It is currently scanning 500GB+ directory, and I’m getting around 15MiB/s.

After various tests and benchmarks, I came to the conclusion that it’s not related to the drive but my OS. So I tested again in Windows 10. Let Syncthing scan the same directory on the same drive and I got 150MiB/s+ this time. I let it scan for more than 5 minutes to make sure it was not the cache giving me a false reading.

---

Computer: Ryzen 3700X, 32GB RAM, Syncthing DB is on an NVMe drive.

I’m running a dual boot setup, but I rarely log in to Windows 10.

My daily driver is EndeavourOS. The drive I’m facing issues is a Seagate 7200RPM 4TB SATA HDD.

2024-04-30 16:56:49 Single thread SHA256 performance is 1794 MB/s using minio/sha256-simd (1794 MB/s using crypto/sha256).
2024-04-30 16:56:49 Hashing performance is 1084.75 MB/s
...
2024-04-30 17:26:55 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29664756310/197196427041 at 15.7 MiB/s (15%)
2024-04-30 17:26:57 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29694116438/197196427041 at 15.7 MiB/s (15%)
2024-04-30 17:26:59 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29729768022/197196427041 at 15.7 MiB/s (15%)
2024-04-30 17:27:01 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29763322454/197196427041 at 15.7 MiB/s (15%)
2024-04-30 17:27:03 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29796876886/197196427041 at 15.8 MiB/s (15%)
2024-04-30 17:27:05 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29815751254/197196427041 at 15.8 MiB/s (15%)
2024-04-30 17:27:07 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29861888598/197196427041 at 15.7 MiB/s (15%)
2024-04-30 17:27:09 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29887054422/197196427041 at 15.7 MiB/s (15%)
2024-04-30 17:27:11 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29920608854/197196427041 at 15.7 MiB/s (15%)
2024-04-30 17:27:13 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29954163286/197196427041 at 15.7 MiB/s (15%)
2024-04-30 17:27:15 walker/movies@0xc0001ba0b0: Walk movies [] current progress 29983523414/197196427041 at 15.7 MiB/s (15%)

[calmh (Jakob Borg)]

The Linux userspace NTFS thingy is not the fastest.

[Nummer378 (Max)]

Linux kernel versions 5.15 and up have an in-kernel NTFS implementation (The Linux NTFS filesystem driver — The Linux Kernel documentation), maybe that’s faster?

(Though apparently the driver can barely write to an NTFS filesystem, so may not be fit for use in your scenario?)

[chamber2038]

While reading about ntfsprogs, I came across the new driver version ntfs3.

Mounted the drive with ntfs3, and now I’m getting around ~30MiB/s. Which almost twice the speed I was getting!

At least now I know what is the root cause. I will have to live with this until I change the FS in the future.

Thank you!