I have a folder with some 300k files, lots of them consisting of a few KB. After adding this folder, synchronization is very very slow. It reads less than 10KiB/s most of the time. I have completely disabled relays on both ends. I have other 2 folders synced, both containing 200k+ files, with more than 10GB each, and it was fast. SyncThing is running on a Synology NAS DS218j.
It is heavy on HDD reads, as can be seen in this screenshot from iotop:
Is there a way to make the syncing faster? It feels like it is writing less than one file per second, which would take weeks to synchronize, but the other folders added previously with similar size took a few days.
According to Synology’s DS218j datasheet, it’s equipped with a dual-core 32-bit ARMv7-based CPU and 512MB DDR3 – not much wiggle room when working with >700K files.
According to your screenshot of top, swap usage was at 547MB (more than physical memory):
The disk buffer was using 212MB, so could try lowering the size, trading slower file access for more system memory, but it’s a bit of a no-win situation.
Although code can’t run while in swap, it would still help to have faster swap. If one of the two USB 3.0 ports is unused, a SSD with a USB 3.0 adapter would have faster read/write speeds than swapping to the internal drives.
I have followed all steps in the optimization guide of SyncThing, thanks for pointing me there! I also bought an USB 3.0 flash drive with 64GB and created a 4GB swap on it, and then removed the 2GB swap from internal drive. Now things look a little bit better, but still not that much. File transfer is now on the order of 5 small files per second.
I also messed with copiers and hashers, tried different combinations of 1 and 2, but the difference was not noticeable. One thing that I did that no one told me to do is that I reduced maxConcurrentWrites to 1, default is 2, but as before could not notice the difference.
The top screenshot is just a point in time, but transmissiond is using up quite a bit of memory (relative to what’s available) and CPU cycles. If at all possible, relocating the BitTorrent to another machine would help (even if it’s ultimately storing files on the DS218j). Also, consider shutting down other services such as Audio Station if they aren’t being used.
You didn’t mention the capacity of the two HDDs and/or how they’re configured (e.g. RAID 1?) and/or being used, but from your earlier description and screenshots of Syncthing’s web UI, it doesn’t look like you need much storage for your Syncthing folders.
If I was using the same Synology box, I’d relocate the files to be synced by Syncthing to a SSD to speed up random access. If it’s within the budget, replacing the internal HDDs, but if not, using an external one. A scheduled rsync job could then in turn mirror files from the SSD to the internal HDDs once a day.
With >700K of files, I’d use a different filesystem, but unfortunately none of the supported ones on the DS218j are ideal with lots of small files so I’d stick with ext4.
At the moment, I have a 6TB Iron Wolf, of which 120GB are synced with SyncThing on a desktop, a laptop and the NAS. One of the shared folders is a Projects folder with 445K files, 30K folders and 70GB, which I use for development, so lots of changes are expected in this folder, and particularly, when I need to do data extraction from the web, or initialize a new Python project, there might be thousands of files being updated or changed in a matter of minutes. I think this one is a strong candidate for an external SSD on the NAS. The other synced folders are machine specific data that I use SyncThing as a backup/sharing tool, these folders don’t change a lot if at all.
If having updates reflected across the desktop, laptop and NAS as soon as possible isn’t a must, increasing fsWatcherDelayS from the default 10 seconds to 60 might be helpful because it’ll reduce the overall number of rescans.
