Option to skip scanning of large files being updated?

I have some scheduled DB dumps and these take quite long time to complete. I noticed how Syncthing is slowing down the process since it tries to constantly rescan the file just to realize it has changed again. The backup storage is slow rotational HDD so Syncthing slows down the overall process significantly by taking ~50% of the IO bandwidth. I tried to play with Advanced Settings / Folder fsWatcherDelayS, but it did not help.

Is there a way to get Syncthing scan to ignore a file that has been updated less then N secs ago (optionally if it is larger than X)? This would improve the overall system performance by eliminating useless re-scans.

Not really, no. I’d recommend having the backup process write to a temp file, then move it to the final destination. run_db_backup > db.backup.tmp && mv db.backup.tmp db.backup. Then ignore *.tmp and you get rid of both unnecessary scanning activity and the risk of syncing partially complete backups.

Thanks Jakob,

This works in case of individual cases and I can live with that.

However, this workaround creates additional complexity elsewhere just for one tool and in some cases this workaround can be rather cumbersome to implement.

Would it make sense to have a grace period for modifications before rescanning (large) files? What kind of drawbacks it could cause?

I understand there can be cases where such a grace period could be unwanted behavior possibly blocking updates “forever”. However, this scan-loop is rather nasty for large files.

I see it kind of in the opposite way. Having long running jobs write incomplete data to temp files and rename when done is always good practice, works with all kinds of sync and backup tools, and has no real downsides. Implementing workarounds in Syncthing introduces additional complexity (extra config knobs to tune, more code, more things to test) useful only in specific cases.

yes, I can understand the edge cases this would create can be nasty to deal with.

After thinking this properly, the way you proposed is a good way to deal with this. :white_check_mark:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.