Best way to configure Syncthing for transferring folder of ~751GB in size, 5.2million files and 207 folders

[MrUser]

Hello,

Just after a bit of advice please! I am using amd64-v1.23.1 of Syncthing, and as per subject I have folder of size ~751GB, 5.2 million files and 207 sub-folders:

On Friday afternoon, I started a sync, and came back this morning, only several folders had been sync’d, no files and Syncthing reported it was still “Scanning”.

I only have one device I am syncing too, and I set the folder Rescan time to 12 hours.

Doing a restart on Syncthing and this made no difference to changing the state of “Scanning”. The console window that opens when Syncthing starts reported “scanning of xyz folder failed”.

I set up one of the sub-folders to sync instead: 121GB in size, 1.4million files and it’s scanning and syncing successfully to the other host.

So my questions are:

1. Is there a hard limit on the number of files and/or size of a sub-directory that can be sync’d?

I did see a post here in the forum: Syncthing file size limit and number of files limit - #3 by Nutomic for one user doing 9 million files and 90TB

This post was from 6 years ago though! Is it still relevant today?

2. What is the best time to set the Rescan time to? Should it always be longer than what the estimated “Scan Time Remaining” first gives on a new scan?

For example the sub-folder which is now syncing, is setup as follows:

As only at 20%, can I assume, 100% = 5 * ~2hr34min = ~14hr50min?

3. Bearing in mind the above, as long as delta updates are going through, is it worth doing a periodic total re-scan?

4. Is there a way of obtaining some verbose output to show what Syncthing is doing when it sits in it’s initial “Scanning” state? I.e. Is it actually scanning or frozen etc.

[calmh (Jakob Borg)]

Scanning many files take a long time. Scan time remaining should be remaining time, so no need for the math you’re proposing. You can get more verbose log output by running with --verbose.

[MrUser]

Thank you for reply @calmh, appreciated.

So I basically just need to be patient and wait I guess? Obviously the bigger the folder, the longer the scan will take?

Used the verbose flag, but unfortunately it does not give any information on the progress of “Scanning”. I see in my logs:

If am syncing three folders for example, do you know if any difference to scan speed is made if I pause syncing on two of them?

[calmh (Jakob Borg)]

With verbose it emits a log line for each file scanned, but it may take a while to build the scan queue first. If I remember correctly, that step is skipped if you disable progress updates, which is faster but then you don’t get the percentage indicator in the GUI.

Other folders syncing doesn’t matter much for Syncthing, unless you’re limited by disk I/O in which case the less happening at the same time the better of course.