Extremely slow initial sync with no obvious bottleneck

I’m running syncthing on two machines: an Ubuntu 16.04 VM and a Macbook. There’s about 5 GB to sync in 110k files. It’s been going on for hours now and is 8% complete. rsync of the same data takes a couple of minutes, tops; didn’t time it exactly but it’s orders of magnitude faster.

Neither of the machines is starved on CPU (single-digit % used on the target Mac, about 30% on the Linux machine) or disk I/O (mostly zero on both sides) but still the network I/O is ~100 bytes per second. The speed seems to be about one file per second.

FWIW, calling /rest/db/status?folder=my-folder-id takes ~4 seconds every time. GUI is closed on both sides. I’m using syncthing v0.14.49

Anything I can do do find the bottleneck? (perf top on the Linux side shows the Go GC and LevelDB having lots of fun but it’s still far away from taking a single core, let alone all 12 threads).

Edit: the hosts see each other over local IPs, without a relay.

Are the files in a single folder? I suspect the database management overhead on small files is causing this.

No, there are about 12k directories (that I could also watch get synced one by one before file sync started).

Can I speed up the DB somehow? I can live e.g. without fsync() after every write if that makes the sync fast (when the DB gets corrupted, I can just nuke and resync, but not really if it takes two days for 5GB).

I mean syncthing folders, not actual folders.

You can’t disable fsync, but that could also contribute to the slowness, I guess you could check with strace which syscalls it spends most time in.

Oh, in that case yes, it’s a single folder.

It’s mostly futex() and pread64(), both are pretty fast though:

https://pastebin.com/qmmMjFPt

Can you run with STTRACE=model env var to see what its doing? Is it actually making progress just slowly?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.