I recently started using Syncthing - thanks very much for it!
I’m currently trialling a deployment across two NAS units, syncing ~4.5m files totalling about 36TB. I understand that this is going to be a challenging environment, and that I need to set my expectations accordingly.
My question is with regards what behaviour I should expect to see, in terms of how the sync process works. My current understanding is that the following takes place, one-after-the-other:
Scanning;
Transmitting indexes across the network;
Requesting needed blocks across the network.
To what extent do these processes overlap? I’m seeing a few anomalies, which might be a result of these steps being out of sync between the two devices, such as:
The global state jumped up by 1m files for a few days, then dropped back down again. Could this be due to file indexes being reconciled between systems?
In reconciling two partially-synced folders (which a previous syncing system gave up on), I initially saw that one was a long way behind the other (further than expected) - but it seems to have been catching up faster than the transmitted data suggests. The scan has finished, so is it just transmitting file indexes and realising that it already has required data?
Scanning and pulling (downloading) are mutually exclusive, but they can happen on parts of the data, that is if the directory is being populated with data as the scan is running, its possible that we’ll only discover parts of data and send indexes for that part, pull some other data, and go back to scanning. If all the data is already there prior a scan, then scan should run until all data is scanned. As data is scanned and added to the index, there is a separate routine in the background dispatching the index to remote devices (you could say in parallel to the scan). As remote devices get the indexes they will start pulling the data from remote device, and the remote device is able to serve pull requests while its scanning or pulling itself.
There are quite a few UX features that make initial scans and downloads on large data sets slow in favor of better ux. One is scan progress interval, other is pull progress interval, these should be set to negative values in the advanced config in your case. (On my phone, can’t give exact names, sorry).
The out of sync status is based on the version of the file, not the data itself, so we might look we are terabytes away from being in sync, yet download nothing when we go into pulling once we realize that the content between the different versions is the same.
It sounds like I should definitely expect some reporting irregularities during this process, whilst the devices get themselves fully aware of themselves and each other. I’ll not worry about it too much for now then!
There are quite a few UX features that make initial scans and downloads on large data sets slow in favor of better ux. One is scan progress interval, other is pull progress interval, these should be set to negative values in the advanced config in your case. (On my phone, can’t give exact names, sorry).
I presume these don’t apply if I generally run with the UI closed. I’ve found that the transfer rate is massively impacted if I keep the UI open, as per this ticket: https://github.com/syncthing/syncthing/issues/867
The initial scan is a far amount of work, obviously from the reading-and-hashing point of view, but also database wise. Each new file gets recorded in the database, and an index update is sent to all other devices (batched). Those devices then pull the file (request its data, which can in parallell with the scanning) and send an index update of their own saying that they have it, which also needs to be recorded.
If both sides actually already have the files, every file is a conflict since they have separate origins. Syncthing will silently resolve it as they are identical (presumably) but this generates further back and forth with index messages and database writes between all devices.
Once all of this has happened things should cool down as only changes get recorded and acted upon.
Do set the rescan interval to something higher than 60 seconds.