Using Syncthing as a Robocopy replacement for one-shot software distribution / mirroring

Hi all,

I’m working on an integration where I want to use Syncthing as a replacement for Robocopy for distributing files from one PC to multiple target PCs. I’m using Syncthing for its benefits of p2p, file block level deltas, etc. I do understand Syncthing is created for continous synchronization, but I believe it could fit my usecase of one-shot deployment / mirroring.

Requirements

I currently have a workflow where one machine acts as the source of truth for a software package, and a number of other machines receive that package.

For example:

  • PC1 = source machine

  • PC2, PC3 = target machines

The requirement is:

  • PC1 is the absolute source of truth

  • PC2 / PC3 should only receive

  • at the end of a distribution run, PC2 / PC3 should be exact mirrors of PC1, so there should not be any extra files on PC2/PC3 when synchronization is done

  • ideally, after the distribution run is complete, the syncing folder should be paused (this replaces an old Robocopy run-once flow, not a continuous synchronization model)

Current implementation

I have a backend application which runs on each PC on the network which stars Syncthing, adds the according list of remote devices to each PC, and the list of folders (which have the devices linked), and starts synchronization.

Issue #1

Synchronizing does not always end with exact mirrored folders

Suppose we distribute from PC1 (Send Only) → PC2, PC3 (Receive Only), given the target folders on PC2, PC3 are not empty.

The expected result is:

  • all files from PC1 end up on PC2 and PC3

  • any extra files that exist only on PC2 / PC3 should be removed

  • final folder contents should be identical

What I observe when syncing is done:

  • PC1 can show all the folders as Up to Date including PC2

  • but PC2 may still contain extra local files that do not exist on PC1, and those extra files are only visible if we inspect Syncthing Web interface on PC2 itself where we get the “Revert Local Changes” button

So effectively:

  • the target device is not a perfect mirror yet

  • someone would need to remote into PC2 and click Revert Local Changes manually

That is the behavior we want to avoid.

Two possible implementation approaches we are considering

Option 1: Keep PC1 = Send Only, PC2/PC3 = Receive Only, and auto-revert on the targets

This is the folder type model that seems safest for strict source-of-truth behavior.

  • PC1 = Send Only

  • PC2 / PC3 = Receive Only

At the end of a distribution run, the managing application would:

  1. poll Syncthing folder status on PC2 / PC3 using:

    GET /rest/db/status?folder=FOLDER_ID
    

    until the folder returns: state == "idle" AND needBytes == 0 AND needFiles == 0 AND needTotalItems == 0 AND pullErrors == 0

  2. detect whether the receive-only folder has local changes / “needs revert”

  3. automatically call the same operation as pressing Revert Local Changes, via:

    POST /rest/db/revert?folder=<folderId>
    

    on the target device itself

Is this the intended / recommended way to achieve strict mirroring with Syncthing? Is there a better signal to poll than just “folder is idle / up to date”?

Is there a cleaner way to know that the target is both: done receiving from PC1, and free of local drift?


Option 2: PC1 = Send Only, PC2/PC3 = Send & Receive, then clean up from PC1 at the end

The second idea we considered is, using folder types :

  • PC1 = Send Only

  • PC2 / PC3 = Send & Receive

The reason for considering this is that local differences on PC2 / PC3 would become visible from the PC1 side more easily, and the cleanup could potentially be triggered centrally from the source side using Override Changes on PC1.

The downside to this is that if PC2 contains extra files, those extra files may get announced to the cluster and PC3 may receive them as well temporarily, until local changes would be cleared from PC1 with “Override Changes”

That makes this approach feel risky for a deployment / mirroring use case, because we explicitly do not want target-local files to propagate.


Issue #2

We do not want continuous synchronization after deployment

This use case replaces a Robocopy one-shot distribution workflow.

That means once the distribution is complete, we do not necessarily want Syncthing to continue rescanning and synchronizing indefinitely. So we would pool again the folder status and then perform some actions. I’m considering one of these options for actions after a successful distribution run:

  1. Pause the folder on the devices

  2. Pause the devices / connections

  3. Leave Syncthing running but disable periodic scanning / only rescan on demand

Let me know if you have any advice and how should I shape my implementation.

I think in your shoes I would script Robocopy or rsync to do this job.

But if you want to use Syncthing for it, Option 1 is the way I would test. I have, in my home environment, a monitoring script that’s in production using the Syncthing API. It wasn’t tough after I figured out how to use jq to parse the responses.

My recommendation if you go down this path would be to test it thoroughly before putting it into production.

Pausing all Remote Devices is a good second best to shutting down Syncthing when you’re not using it.

Thanks for the reply, I will continue to implement and test option 1.

I think Syncthing is the best solution for now, as the current Robocopy script that we use has proven to be unreliable and network bandwidth is an issue sometimes. Installing WSL for Rsync on each node would be too much of a hassle.

I suggest you try FreeFileSync. It should be much simpler to install/manage, and it will give you logs of what it has done. You can set up custom rules for handling copy/update/delete, and you should be able to script the setup. I think it will be much more satisfactory in this application.

1 Like

Hi,

I’m pretty sure you can solve your problem easily with a small app called rclone. it is like robocopy or rsync on steroids. You can connect to pretty much any kind of drive you need to. Then

Rclone sync a:/software b:/software

That’s all there is to it. I can assist you setting up the config if you need help.

you can create a batch file that does the file distribution synchronization and either run it manually or put it in a task scheduler.

You will be able to copy or sync files from A to B&C.

It won’t be bidirectional like syncthing.

If you’re running on Linux don’t install rclone from your Repository because it will be an old version go to their website download page and there is a Linux script that will download and install the latest version.

Actually you’re running on Windows so the download page has a zip file for Windows and you just extract the one file called rclone.exe

run “rclone config” to set up a remote.

2 Likes

Yes, https://rclone.org is also an excellent recommendation for your use case.

1 Like

Thank you for all the replies, I took into consideration all the other apps before choosing Syncthing. The main reason for which we decided to walk away from robocopy was the network bottleneck. Sending 60GB (aprox 300k files) over network from 1 PC to 10 other PCs kept failing and was painfully slow. Syncthing was chosed for P2P transfer, which should address this issue. From what I got to understand free file sync and rclone do not use p2p. Only Rsync does, but we cannot install WSL on all PCs, it would be too much extra complexity.

There are definite benefits to the distributed nature of syncthing and it excels at reliable transfer over large distances in my opinion. We also use it for one way transfer but our use case is a bit different.

You can use it but I think it will require some scripting.

You could perhaps create some files for signaling which a script would use to know what to do… For instance. Share a top level folder called deployments.

Then underneath create a folder for a new deployment with all the “stuff”. Create a text file of the hash of all the stuff in that deployment folder. All remotes get all the files including the hash. Alternatively deliver a deployment as a zip file. Once the file is present on a remote, unpack it in a temp folder and install it.

once each remote installs the package, it writes an empty file with its machine name. This file also gets synced to all remotes and the host but you don’t care.

The host monitors the folder for all the empty files. Once all the empty files are created the host knows all the installs were successful and it deletes the package, removing it from all remotes.

There are ways to do what you want but you’ll have to write some helper scripts.

1 Like

Correct, they’re client/server without directory services for locating other hosts.

There are native ports of rsync for Windows, so WSL isn’t necessary.

If encrypted transfers aren’t required, then running rsync in daemon mode on the target PCs would provide a simple one-shot solution where PC1 can invoke rsync as needed.

Other related solution is Unison.

1 Like

Because PC1 is send-only, it won’t receive local files created on PC2 and PC3, and unless PC2 and PC3 are peered, files on PC2 won’t appear on PC3, and vice versa, so option 2 is simpler to maintain than option 1.

Hi,

When you say you don’t want to put Windows subsystem for Linux on a lot of computers it’s seems contradictory to the master server to the two slave servers concept.

Rsync technology is very old and I don’t know anyone who uses it anymore but here below are some options for you to choose to create an rsync server on Windows. Last option.

I still believe that rclone sync files is the most reliable way to go. If it gets interrupted it will pick up where it left off.

Rsync for windows or at least a port is available here:

rsync from samba

rsync is also included with

Cygwin

A native package is here:

cwRsync