Any option to sync 500k small files in less than half a day?

What are your performance guys? I am trying to figure out whether it’s on my end or is syncthing just slow af no matter settings?

If you have a desktop app try syncspririt, it shows order of magnitude faster on a small files.

It’s a sore spot; I have an experiment with database sharding that speeds it up a lot. As-is, you can gain a couple of magnitudes on the receiving end, for lots of small files, by disabling fsync and block indexing for the folder.

With those settings, syncing 500k unique tiny files takes about 20 minutes on my laptop.

To illustrate, here’s a graph of three runs I did with different settings. It’s minutes runtime on the X axis and number of files synced on Y. With default settings you’re unfortunately on the purple line, which I stopped after 15 minutes an 60k files. Turning off fsync and block indexing puts you on the green line where it finishes after 17 minutes. Also enabling case-sensitive-filesystem gets you to the blue line where it finishes after seven minutes.

It kinda sucks, but these integrity/safety settings come at a high cost.

(The orange line is an experimental branch which happens to cheat by starting earlier, but also keeps a steeper slope so that’s nice.)

I mainly need to sync windows ntfses - i suppose this bench is on linux + ext4?

second thing - turning of fsync will just risk receiving side for data loss (power/crash etc)? I mean - there is no way it will return back corrupted data back to sender? If yes it’s a risk worth taking :slight_smile:

This was on macOS, but yes. Disabling fsync is safe unless you have a sudden power failure. Disable case insensitivity checks is in principle unsafe, but if you know your source data doesn’t have any case conflicts (like, it’s already on Windows) then it’s safe for at least a one time sync operation.

I also just found a way to make those checks a lot faster, so improvements are coming there.