Okay, background on the rename detection mechanism:
Syncthing scans for changes. The filesystem notifications we get (when enabled) triggers the scanner.
The scanner traverses the thing it was told about. Sometimes this is a single file, sometimes it’s a parent directory, sometimes it’s the entire folder. Notification events are aggregated so if we see more than a threshold of events in a given directory we scan the entire directory in question.
When scanning we detect new or changed files. Those are read and hashed.
Once we’ve hashed a file we take a peek in the database for files with the same hash. If there are matches we check if those other files still exist. If they don’t, we can realise this is a rename and pair the addition+deletion so the other devices will also understand it’s a rename.
Once we’ve done a full scan for new and changed files we look for deleted files – that is, are there files in the database that are no longer on disk? If so, mark them deleted.
So my best guess on root cause is that if a scan triggers while files are being moved one by one, there will be further moves happening during step 3-4 and those will be flagged as deletes only in step 5. Then we’ll discover the target files on the next scan, but by then it’s no longer a rename but an addition.
If this is the case, increasing fsWatcherDelayS and fsWatcherTimeoutS to cover the entire time the “rename” takes should mitigate this, and indicate this was the problem. (Or simulate the same thing by turning off notifications and trigger a manual scan after the entire move operation.)
I select hundreds of dirs and move them to a different subfolder (using F6 on Total Commander - I guess its like Cut + Paste on Windows Explorer).
So I;m guessing each one is moved separately. The whole action takes like ~1 second, but each one gets it’s own treatment I’m 99.9% certain.
But the scan is far from being triggered, there’s plenty of time ahead, I have to wait another ~9 seconds for it to be triggered.
When I move only one or 2 folders, it’s instantly synced (10 seconds later).
But when I move 100s, it’s not. It goes into a sync/scanning & then delete & copy on Bob.
The scanning on Alice, I don’t mind so much, although it can take a couple of hours. But the re-transmitting kills the whole utility, as it can take many more hours (and days if connection is remote)…
I’ll take a look at this. Likely I haven’t changed these from the defaults. I wonder if it makes sense if watcher is suddenly being flooded with changes that Fswatcher waits until the changes cease. Anyway I will try adjusting these.
As another poster mentioned the subsequent sca can take an hour or two if 500GB has been moved or renamed.
This is done on the source side? Or this hash check is done on the destination side? And this is done DURING the scan? Or after the entire scanning is completed?
And this detection would be successful if the file was moved into a different SUBfolder AND renamed?
The pairing of the create+delete events is done at the destination (logged as ‘taking rename shortcut’ if logs enabled’) though i guess there’s a bit of both as events are carefully batched to allow that? (pls correct me if I’m wrong Jakob). We’ve a really hacky workaround of deferring the deletion scan until the previous scan completes without any errors. It works -most- of the time, with some edge cases.
Another possible workaround could be to skip the “look for deletions” step if we found at least one rename in the previous step. That’d probably fix the problem in this case, but would delay seeing actual deletes if they happened in the vicinity of renames.
Or simply move deleted files to a temp .stcache (or something) place in Bob, until we’re sure those were real deletions in Alice. If they prove to not have been deletions (i.e they were merely moved in source Alice) move them to where they should go in Bob also. This way files will seem as they have been moved/deleted in Bob instantly, but they will be available to be picked up without need for retransmission, if Alice simply moved them. If Alice really deleted them (after finished scanning/hashing), delete them from Bob’s .stcache. Consistent and efficient.