I have been trying to benchmark Syncthing for various kinds of files. I was trying to simulate creation of 10000 files of 10KB each. The transfer speed comes to around 160KB/s. However, when I tried with files of 100MB each, the transfer speed came to around 16MB/s.
I read through some of the answers in the forum, which suggested to increase the number of pullers/copiers to increase the speed. But that did not result in any changes at all.
I set the pullers to 150 and copiers to 50.
One more issue is the deletion of files, which approximately takes the same amount of time as syncing (around 10 minutes for 10000 files of 10KB each). I’m not sure if this is expected behaviour. Any help will be appreciated.
P.S. I’m running syncthing as a container, with 1GB of memory and 1CPU. Not sure if this would affect the application in any way.
It’s expected-ish. Syncing a file involves creating a temp file, copying any still valid data it, downloading new blocks to it, flushing and syncing that file, deleting the old file if there is one, and renaming it in place. Many if these operations are slow at the filesystem level. The OS and filesystem impose a limit on us here in terms of file operations per second, which becomes relevant for lots of small files and irrelevant for larger files.
Thanks for the reply. Could you elaborate a bit more, or point me to the right resources as to what kind of limitations the filesystem/OS imposes on file operations.