FS Watcher suggestion - batch change interval

Feature
1

With 0.14.47, there’s a new feature that enables FS change watcher at syncthing. This is great. So now as soon as a change happens, it gets propagated. Definitely a huge improvement over waiting for an expensive full scan.

Can I have a suggestion for advanced users: can we have a middle ground option between full scans and filesystem changes? That is: all FS changes are still sent as changes to the share, but the changes are sent at a different user-configurable interval, a “batch change interval”. Before then, changes are queued.

For rapidly changing directories, propagating share changes with each change can make the network and disk noisy for other nodes. Frontends such as syncthing-gtk as well get noisy on the notification side. Obviously with a small number of nodes / changes this is not an issue. Imagine, however, saving / regenerating a 1GB video a few times on a 3-node share. You can easily be trying to force 10+GB into the network in a span of 1 minute. It would be helpful, though, if syncthing batched the FS-detected changes, say, in 5 minute intervals. Come the batch send time, it could then send all the changes, skipping over effectively superfluous ones (e.g. if you change the same file 5 times in the batch interval, only the last version effectively gets sent, dramatically cutting down bandwidth / disk usage).

The way the FS watcher works right now is effectively the same as what I descibed above, except with the “batch change interval” set to 1 second (or whatever the granularity of the FS watcher is), i.e. all changes are sent effectively immediately. But it would be nice to have a switch to control the interval to say, every 5 minutes or so.

This is different from a simple rescan in that it doesn’t force a re-read of the share again: the changes sent will simply be the ones previously detected by the FS watcher. For larger / less performant shares the FS watcher is a huge bonus over a rescan. You can still set a rescan interval to maybe every hour (depending on the share). Obviously there’s not much sense in having a “batch change interval” larger than the “rescan interval” since a rescan is a more comprehensive form of checking for FS changes.

Anyways, loving the new syncthing. What I’m asking for actually sounds like a hard problem (not sure, but you have to have a change queue and a way of calculating which changes in the queue are superfluous) so I understand if it’s a long shot.

1 Like
2

There is a default timeout of 10s (change will not be withheld for more than 10s or some other heuristic) that is adjustable. Worst case you can go to the old method.

3

Is this timeout configurable? Didnt see it in the web interface

4

Its in the advanced settings, not in the generic folder settings.

5

It’s not on 0.14.47.

syncthing-advanced-edit-folder
syncthing-advanced-edit-folder.png898×512 26.4 KB

There is a checkbox for “Watch for changes” and an interval for “Full rescan”, but no timeout / interval for the fs watcher. If there is one, I’d like to set it separately from the full rescan, for example, setting the FS watcher to update every 5-10 minutes, and then setting full rescan on the interval of 12+ hours.

6

Settings -> Advanced -> Expand the folder you care about

7

I see. I confused that with the “Advanced” tab on folders. It’s kinda confusing that they aren’t the same though.

8

Before we had tabs in the folder editor, there were no space to wedge in all settings.

9

I assume it is the “Fs Watcher Delay (seconds)” setting? Any hard limits on the value?

10

Yes. I believe it’s an int, so depending on your platform it’s either the 32bit or 64bit limit. What’s your use case though? In my opinion for very long delays (something like >1h) it is quite useless to use the FS watcher, you could just do a full scan at those intervals instead.

11

Hi Simon! Danke for the quick heads up!

The reason for prologing the FS Watcher Delay is just fine tuning the efficiency of Syncthing on my end. Certainly not an everyday use case, but one which FS Watcher enables me to do.

Without going into the gritty details: I am syncing environment data from - say “weather stations”. Those stations record their data every 10 mins. they are running on a low energy platform, think “raspberry pi”. Writing the data to (a zillion files) the internal memory storage takes 17 seconds.
Before FS Watcher I had a sync interval of 24h (86.400 secs). This scan takes about (don’t laugh) 40 minutes. That is why it is done so seldomly.
With FS Watcher I still have that 86.400 sec full scan interval, but I can sync the data in shorter intervals. I’m just waiting until the write process is done.

Thanks for madumlao for asking this in the first case!