I am trying to use Syncthing to do a one way copy of growing video .ts files from 20 laptop recorders to a single server over a LAN. It works sporadically while the files are being recorded and logs hash errors because the file changed while it was generating the hash. I am looking for a way to have it sync more regularly without having to stop or segment the recording.
I have workflow reasons for needing to copy growing .ts or .mkv files. I am open to education on the technical difficulties of this use case. I am guessing industries such as the security industry have tricks to make something like this work. I am receptive to alternate tools or services that will help.
Syncthing could work with this if you set up some kind of a script that would periodically (e.g. once per hour) create a copy of the video file in a separate folder that would be then synced. In other words, don’t sync the main video file, only the copied version that doesn’t change and thus can be hashed in full. Obviously, doing so would require twice as much space on the disk.
During the recording video editor will be reading but not writing the growing .ts files on the server. At the end of each day we stop the recorders and intend to have an identical duplicate copy on the server.
Even if you get Syncthing to work somehow, it’s going to be wildly inefficient because the entire file gets rewritten on the destination every time it gets updated. I think you should find something else that just streams the file there concurrently somehow.
In that case, it sounds like the --append option in rsync might be a viable solution. It can only be safely used on files that only grow in size and when the receiving end isn’t being modified by some other process.
The caveat is that rsync isn’t a continuous sync tool, so it must be run by a script in a loop or a scheduled task. But since rsync doesn’t rehash the data that’s already been transferred, it’ll be slower than a network file share, but considerably faster than the default rsync mode, Syncthing, rclone, etc.
Since the laptop recorders and server are all on a LAN, have you considered using a distributed filesystem or something along those lines?
Laptops are currently Windows 10. I have not seriously considered a distributed file system. We hope to migrate the workflow to sync with the cloud during phase 2 of our project so I will have to plan for that as I implement this.