Windows one direction one shot syncing task scheduler

Hello. New here. Will try to make this short. My windows scripting skills are very limited. Below scripts were created using samples I found online but need improvement.

I have two instances of syncthing installed.

a) client - windows pc b) server - freenas box

Goal: To periodically (once a day?) mirror files from the window pc to the nas. The whole process should be automated.

I’ve broken this down into two parts.

  1. Launch syncthing process on the windows pc at bootup and/or login
  2. Perform sync at scheduled time(s).

Initial launch at boot/login

Using windows task scheduler with triggers at startup, at logon, work station unlock, the following script will run.

start.bat

@echo off
SETLOCAL EnableExtensions
set EXE=syncthing.exe
set LOG=E:\syncthing\log.txt
set home=E:\syncthing
set PROGRAM="E:\syncthing\syncthing.exe"

rem syncthing start loop

FOR /F %%x IN ('tasklist /NH /FI "IMAGENAME eq %EXE%"') DO IF %%x == %EXE% goto FOUND
echo  %date:~4,10% %time% STARTING SyncThing Process  >> %LOG%
start "SyncThing" %PROGRAM% -no-browser -no-console -paused -home=%home% > NULL
goto FIN
:FOUND
echo  %date:~4,10% %time% SyncThing already runing. >> %LOG%
:FIN

The script checks to see if syncthing is running. If it is, it does nothing and exits. This gets the job done, but I’d to add another check to make sure syncthing is responsive (responds to the a ping via api?). If output is pong, then exit, otherwise kill and restart the process.

If process it not running, it starts it in a paused state.

The next part is the actual mirror process initiated by task scheduler. Trigger will be a specified time.

During testing I discovered if both sides are not paused then erroneous log entries get created with one side trying to communicate with the other. Pausing both ends resolves this.

The logic is unpause both sides, wait a minute, perform mirror, pause both. Although this will be used on local lan only, https used in case some other program happens to be sniffing network :).

Any suggestions for improving this task or implementing differently? Variations of this batch file will be used for specific folders (some only need to get sync once a week) using optional parameters - POST /rest/db/scan — Syncthing v1 documentation .

sync.bat

@echo off
SETLOCAL EnableExtensions
rem define variables
set LOG=E:\syncthing\log.txt
set local_apikey={API KEY}
set remote_apikey={API KEY}
set local_deviceid={DEVICE ID}
set remote_deviceid={DEVICE ID}

rem Unpause source and destination services

echo  %date:~4,10% %time% Unpause local and remote servers >> %LOG%
curl --insecure -X POST -H X-API-Key:%local_apikey% https://localhost:8384/rest/system/resume
curl --insecure -X POST -H X-API-Key:%remote_apikey% https://192.168.1.11:8384/rest/system/resume?device="%remote_deviceid%"

Rem perform sync

timeout 60 /nobreak > NUL
echo  %date:~4,10% %time% Perform sync  >> %LOG%
curl --insecure -X POST -H X-API-Key:%local_apikey% https://localhost:8384/rest/db/scan
timeout 5 /nobreak > NUL

echo  %date:~4,10% %time% Pause local and remote servers >> %LOG%

rem Pause source and destination services
curl --insecure -X POST -H X-API-Key:%local_apikey% https://localhost:8384/rest/system/pause
curl --insecure -X POST -H X-API-Key:%remote_apikey% https://192.168.1.11:8384/rest/system/pause?device="%remote_deviceid%"

Thank you for your input.

Hello! Your wrote:

Goal: To periodically (once a day?) mirror files from the window pc to the nas. The whole process should be automated.

The goal for syncthing, as stated on the top of syncthing.net, is:

Syncthing is a continuous file synchronization program

These goals do not match. Why not just let it do its thing? Just fire it up and forget about it.

2 Likes

Because that is not my goal :thinking: I don’t want continuous or interval based syncing.

I thought about using the scan interval but that wouldn’t cause scans to happen at specific times. There is no built in scheduler so must use tools available to do what I want it to do. The software is flexible enough. With a little elbow grease it will work how I want it to. :+1:

I don’t use Syncthing like that, but the concept is interesting. This looks like what @Catfriend1 has been doing in his Syncthing Fork for Android.

I assume that curl is a 3rd party utility here, right? You could use PowerShell (see https://forum.syncthing.net/t/how-to-call-the-rest-api-from-windows-powershell/11269) to query the REST API instead to make the script more universal and not rely on any external tools.

How are you going to kill the process? I only want to warn that it is impossible with taskkill without the /f switch, which I would not recommend, as it is a very brute-force solution and may lead to issues.

Killing the Task Scheduler task itself also doesn’t work, because it only kills the parent process, while the child process persists (see https://github.com/syncthing/syncthing/issues/7042).

1 Like

Why aren’t you just using rsync? Seems like a lot of machinery for something simple.

1 Like

Or Unison for a two-way, windowsy option.

Curl is a built right into windows 10. So is an ssh client. No additional tools or utils were installed to get that initial batch file going.

Good question and point. I suppose a graceful shutdown would not be possible anyway if the service is unresponsive. One could try an api call to shut down first but if the ping call is unanswered I suspect a shutdown request will result in the same.

It was a consideration. I don’t believe there’s a pause/resume ability with rsync. Once started it needs to complete. Having gui access is nice too.

I suppose the main difficulty is in initial configuration. This of course applies to both, but is more difficult with rsync. Maybe you can point me to a good guide that describes how to set up this type of scenario? In searching I haven’t been able to find anything specific (receiving end is a freenas box).

Sync intervals will vary for specific folders. Some daily, others weekly. GUI config is much easier than commandline.

I realize using ST is overkill for the task but it seems like the best solution given my skill level.

You use pause/resume syncthing to start/stop the process of syncing. The equivalent in rsync, is you run rsync to sync files, and don’t run rsync to not sync files (amazing, isn’t it?).

Rsync does not need to complete, and is fine to be interrupted at any point. If you restart it, it will re-do some of the work it did previously (to work out what changed etC), but after that it will continue (on a per file basis, it either syncs whole files or nothing). It’s stateless (so it does not have a database or anything you need to care about).

Rsync requires zero configuration, it’s literally, run a command:

rsync ./my-folder username@remoteip:/some/remote/folder

Done.

I am not sure what sort of information you are looking for, but rsync comes with a manual page on most distributions. Also I am not sure what you mean by “set it up”, as it’s literally running the command I’ve written out above, just like you run you batch script. Sure, there are some extra flags if you want to perform deletions, or preserve timestamps, but that’s all in rsync docs/man page.

Sure, when you have a hammer, everything looks like a nail, and we’ve seen those users before.

There have been cases where users post their usecases, where I’m sitting there thinking “Do you really need syncthing here? Seems like a periodic copy files from A to B, that you could probably run rsync on a cron to achieve, much cheaper, faster” etc.

But this is the first time where I see a literal reimplementation of rsync in syncthing.

Don’t take this as criticism, I understand there are only good intentions behind this, as you are just solving a problem that you have, and the intent of this post is to help out others who have a similar usecase.

But from the perspective of someone who took part in writing the software and understands the sort of gaps in existing tools out there that syncthing tries to cover, cases like this look completely mad.

You are right. However, this seems to only be the case in Windows 10 v1903 and later. Not that it matters in your situation, but it could be something to think about if the script was supposed to be more of a general-purpose tool, intended to run on various Windows machines.

Pedantically speaking, it’s also not curl it’s “just” an alias for the PowerShell command Invoke-WebRequest so the parameters and functionality are quite different. :slight_smile:

That is different. curl is an alias inside PowerShell, but now it is seemingly possible to also use curl (or curl.exe) to have the actual cURL (which works in the basic CMD too).

The details are at https://techcommunity.microsoft.com/t5/blogs/blogarticleprintpage/blog-id/Containers/article-id/16.

Ah, cool, I didn’t know that. So I guess you get a different curl depending on which shell you’re in…

(I like how the sample video on the link (of tar) uses it to create and extract a “zip” file. :slight_smile: )

Yeah, I also only got to know this thanks to this thread. I normally have access mainly to Windows 10 Enterprise/Education machines, and those are mostly still v1809 or older, so these tools are not available there yet.

My guess would be that you can still use the “real” cURL in PowerShell by typing curl.exe explicitly.

Good to see we got the curl business figured out.

After exploring other options I still keep coming back to syncthing for my use case for ease of use, gui access, progress reporting, and some other benefits.

Any suggestions on how to implement the checks I was looking for into the batch files in the original post? That is, code to confirm syncthing is responsive.

I’m not sure what you’re asking, really. If Syncthing responds to API requests it’s responsive?

^^I think that’s one way to determine - unless you have other suggestions. I need some guidance on how to read this in a windows batch file please.

Do you mean something like this?

powershell -Command "& {Invoke-RestMethod -Uri http://localhost:8384/rest/system/ping -Headers @{'X-API-KEY' = 'wg9ueKKcPqGu356wTskrQRNNXSqZDSNp'}}" | >nul 2>&1 findstr /l "pong" && echo;Synching is responsive.

Maybe. I will test this out once I resolve the following issue.

Testing with some large 6-10 GB files where slight changes are made in near the end (email file for thebat!). It syncs to the receiver, but generating a sha256 on both (source and destination) results in mismatch. Test is simple. Open program, send email to myself, receive it, close program. Perform sync, test sha256.

Also, file sizes differ slightly.

source 6,847,185,543 bytes target 6,847,181,844 bytes

This is troubling. If the file doesn’t exist then a perfect copy is made.

Turned out to be a path issue. The path in question is really long and 6 or 7 levels deep. It was updating a path, but not the one I was checking against :man_facepalming::man_facepalming::man_facepalming:!

All good after fixing this stupid mistake. Serves me right for doing this after midnight.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.