This topic has come up a number of times. One common use case of Syncthing is to act as a component in a larger automation system that performs a one-way (or mainly one-way) sync of files from a remote host to a local machine.
As a part of this automation it might be desirable to trigger further processing of these files upon completion.
As this topic has come up in the past, one possible solution is simply to poll the file system for completed files. In a low-power system, however, we may want to avoid unnecessarily spinning up the disk simply to check whether there are any available incoming files. The response at this point is typically to take advantage of the long-polling events API.
As I set out to do exactly this over the past weekend, I started to realize that this is a bit easier said than done - primarily with regard to the fact that you may have an unreliable consumer of the API. To build a truly reliable system, your API client must contend with a number of issues. To name a few:
For one, you need to figure out how to checkpoint - as replaying the stream of events from the beginning of the trim horizon (ie, the oldest event in history) may not be very practical. Secondly, we must also contend with the fact that the majority of the events are stateful in the sense that they are dependent upon a particular state of the filesystem at particular moment in time. For example, the FolderComplete event may only be valid at and shortly after the moment in which the event is emitted. Additionally, there are locking/synchronization issues where an ItemFinished event may occur, but be invalidated by additional changes from the remote peer before the post-processing script has completed processing the file.
As it stands, it appears to me that the events API is most suited to be consumed in real time alongside a mechanism for dealing with a full state refresh and a healthy amount of error checking. While all of this have well known, stable solutions, we are inching into a fair amount of complexity to accomplish even fairly simple automations.
I am writing to inquire the appetite to introduce direct support for post processing synchronized files within Syncthing. Many issues above can be addressed in a far simpler manner from within the Syncthing process as it likely already has an accurate representation of file system state. If such an appetite does exist for introducing first-party support of post-processing scripts, I would be more than happy to introduce a PR for review. Otherwise, I am welcome to discussing alternative methods to supporting these use-cases - perhaps more APIs or a reference client that implements post-processing behavior?