Extremely slow resynchronization/reconciliation with Syncthing v2

I’m using Syncthing 2.0.12 on a bunch of reasonably modern machines (slowest is i7-8550U, 16 GiB RAM, rest are workstation-grade laptops, plus a very beefy Zen 3 personal server). All machines use NVMe SSDs with ZFS on them. How do I speed up resynchronization of an extremely large folder (~80 GiB, 2M files, 300k folders) which is already physically present on all devices?

The reason for resynchronization is disaster recovery on one of the laptops: the Syncthing database was reset, mtimes/ctimes of all files in the target folder were changed, but the data has been copied with rsync in advance.

I’m using Syncthing 2.0.12 on all devices and it feels like this process has gotten severely slower in the 2.0 branch compared to 1.x (after the SQLite migration). In short, I am observing an obscene amount of CPU time usage and disk I/O on the target device without making almost any reconciliation progress.

I can note that the hashing finishes reasonably fast (I can confirm this if I pause remotes and let it scan in isolation) — consistent with the amount of data in the folder. However, once I resume connections, Syncthing on the target keeps reading and/or writing something at ~100 MiB/s for multiple hours, with the total amount of I/O clearly not commensurate with the size of the folder or the database, and with no end in sight.

What should I do?

Its probably caused by ZFS, as it doesn’t (always) play nice with SQLite: SQLite database doesn't scale well to very large setups · Issue #10264 · syncthing/syncthing · GitHub

Support bundle or memory+cpu profiles might clarify the details.

Interesting, thanks for the pointer.

For the record, I’m not seeing any issues with ZFS & Syncthing on several servers I manage.