Large files sent through Syncthing have very slow reads from disks?

I’ve been using Syncthing for awhile and recently came into a serious problem which is basically going to require me to wipe a 100TB backup server. I noticed most of the files I’ve synced in the last few months have very low reads from the disks (~11.7MB/s). It seems that it might only affect very large files (50GB+) but I haven’t been able to confirm. However, every single 50+ GB file is affected.

  • Around 400 50GB+ files I sent through Syncthing can only be read at ~11.7MB/s from the disks on the backup server.

  • 22 different disks are affected, so it’s not the disks themselves.

  • Files on the disks that were not sent via Syncthing read at 150-200MB/s.

  • Transferring the data off the disks at ~11.7MB/s, deleting the slow files, and then transferring it back via SMB/Windows results in 150-200MB/s reads again.

  • The disks are not splicing or sharing data and are on 3 different SATA controllers.

  • Both the main server and backup server are on unRAID and using XFS filesystem.

The only thing I can think of is Syncthing is causing extreme fragmentation when dealing with large files however I feel like it wouldn’t be a consistent ~11.7MB/s if that were the case. This honestly is making no sense to me, so any ideas why this could have happened would be helpful.

Syncthing transfers files in chunks, in a random order. If you are using spinning disks, the resulting fragmentation will be quite annoying.

I don’t have experience with XFS, but it seems like there are tools for measuring fragmentation, and for defragmenting (found with google):

Relevant prior discussion with some advice:

I’m going to try what is suggested on that thread, enabling “Large Blocks” and setting Copiers to 1. Thanks.

The blocks will still be delivered in random order they arrive, but yes, large blocks will help.

My understanding is that fragmentation should not happen at the disk level, as I believe spinning disk seeks to sector representing that offset to write data.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.