One-Time synchronization with constant file changes

Support
1

Hi all,

I have a Production machine with constant file changes and a fail-over machine that needs to be the most up-to-date possible.

I need to use Syncthing in two different scenarios:

  • Initial synchronization
  • Post-Initial synchronization

The Initial synchronization scenario is where I need to do something similar to a backup/snapshot from the Production Machine (let’s call it SourceFolder) to the Fail-over machine (let’s call it TargetFolder). After completed, I need to disable it and switch to the Post-Initial synchronization phase.

The Post-Initial synchronization scenario is the standard use of Syncthing, synchronizing any changes between SourceTempFolder and TargetTempFolder.

Summary:

The problem is that since SourceFolder is getting constant file changes, Syncthing will probably never reach 100% sync, and consequently I cannot tell deterministically that I can switch to the Post-Initial synchronization scenario.

What I was trying:

  • Oldest First
  1. Identify the most recently modified file (call it FileX) in SourceFolder
  2. Set SourceFolder as Master and synchronizing oldestFirst
  3. Monitor the TargetFolder to see when the FileX is created and them stop the sync

The problem is Syncthing transfers multiple files at the same time and detecting the FileX doesn’t mean that all older files are already there too. Other problem is that Syncthing creates .tmp files that will stay there forever after stopping the initial sync.

  • rescanIntervalS
  1. Identify the most recently modified file (call it FileX) in SourceFolder
  2. Set SourceFolder as Master and setting rescanIntervalS to 0
  3. When adding the SourceFolder to Syncthing it will scan once (like a snapshot) and start synchronizing
  4. My logic can look only for changes that occurred after FileX modified date and copy files from SourceFolder to SourceTempFolder

During the synchronization files may be deleted/modified, and since rescanIntervalS is set to 0 will Syncthing have problems?

Do you guys have any suggestions?

Thank you in advance, Eduardo

2

I don’t understand the difference between initial and post-initial synchronization. It seems both cases are the same, except TargetFolder is initially empty?

3

The difference is that the SourceFolder is constantly changing and by consequence Syncthing will never reach 100% sync with the standard configuration and I need to be able to stop the “Initial Synchronization” phase at some point and switch to the “Post-Init” phase.

I need to tell deterministically that every file with a modified date lower than X is synced, and the Post-Init phase will look for changes after that X date.

4

I’m still not entirely clear on why you need this, but it sounds like it may be simpler for you to use something else, like rsync, for the initial one shot sync.

5

Basically I want to track and react to changes in the Post-Initial Sync phase. In the Initial Sync phase I just want to replicate what’s in the SourceFolder to the TargetFolder and them stop the sync.

Regarding rsync, I would like to have 1 solution for all phases, as well as present the sync status on a WebUI.

Do you have any idea when/if this enhancement will be implemented?

This could help me by configuring the “Initial Sync” folder to only sync files older that X date.

6

That is a long term ticket. While a very popular feature it is not simple to implement. I would not expect it soon.

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.