Increase the number of threads used by the External Versioner?

I have been using the external versioner to send files to the Recycle Bin in Windows for a few months now (see https://docs.syncthing.net/users/versioning.html#move-to-the-recycle-bin-using-powershell).

I have observed that Syncthing spawns at most two PowerShell windows at a time, and then spawns new ones only once one of the two previous ones has exited. The problem with this is that a) PowerShell itself is rather slow to start, so it always takes some time to spawn the process, and b) even without considering the PowerShell’s speed, it is still slow to process a large number of files this way.

Is there any way to increase the number of these processes allowed to run at the same time? Now, I need to disclaim that I haven’t analysed the code responsible for the external versioner, but is the number of copiers perhaps involved in it in any way here? Would increasing copiers help the external versioner in spawning a larger number of processes at a given time?

It’s probably a good idea to look for a native utility and sidestep powershell: GitHub - sindresorhus/recycle-bin: Move files and folders to the Windows recycle bin

1 Like

I would prefer to stick to built-in solutions only, but regardless of the tool, Syncthing still spawns only two instances of the process anyway.

I’m going to sync around 200,000 deletions very soon, and I have temporarily increased the number of copiers for the folder in question, so we are going to see whether there is any difference or not.

I’d also try to avoid output in the shell.

I can now confirm that the number of copiers does not seem to matter. From what I can see observing the spawned processes, Syncthing seems to simply version the files one after another, without any actual multithreading.

On fast hardware/storage, this is not a huge deal, as still a couple hundreds of files are managed to get processed per minute, but on slower machines, it can take a long while to go through all the files.

Versioner stuff happens in the finisher routine, of which there is usually only one, so I am surprised you’re even getting 2 windows.

Unless some of your operations are renames, which then avoid going via the finisher (hence could open a second window).

This number is not adjustable, and I am not convinced it would be safe to make it adjustable.

I guess that it may only seem like it in the Task Manager, which probably has some kind of a refresh delay, while in reality Syncthing just keeps opening only one new window at a time.

I do understand what you mean, although it would probably be safe in this case, as Windows gives a unique identifier to each file that goes to the Recycle Bin. Anyhow, so I guess that I will have to think about how to make each process open and close as fast as possible to make the queue move faster…

I am not concerned about the safety from windows side.

I am concerned about something going bad syncthing side. Can’t think of anything off the top of my head, but I am not sure that it would all be good either.

I will probably implement a two-step solution.

  1. First, call a one-line batch script just to move the file to a temporary folder on the same partition. This should execute instantly.
  2. Then, schedule a PowerShell script to run every hour or so and move all the files from that temporary folder to the Recycle Bin in one shot.

This should be much faster, because cmd.exe has much smaller footprint than powershell.exe, and also calling PowerShell on a folder with multiple files only once will be much quicker too.

Seems to me that you’d just enable trash can versioning in syncthing and just run your recycle bin thing against syncthings version directory?

No need to spawn cmd that just does what syncthing does.

1 Like

If you want it fancier, you can even run something constantly, listening on folder status events, instead of hourly, to move the Syncthing trash can to the windows trash can whenever Syncthing finishes syncing.

Yeah, although I would prefer to keep everything in a single folder (for the PowerShell script), and the default versioning will likely overwrite files with the same names coming from different Syncthing folders. Also, there is a different problem that I haven’t considered yet, i.e. recycling files from their non-original location will make restoring them later much more difficult (as everything will go back to the temporary folder instead of the original locations).

I probably need to think about this more in order to come with a decent solution that will be lighter on the resources and definitely quicker, as right now it may take hours or possibly days just to proceed all the versioned files.

I haven’t done anything like that myself yet, but PowerShell seems to have such functionality (e.g. https://mcpmag.com/articles/2019/05/01/monitor-windows-folder-for-new-files.aspx), so this may be handy indeed… I will need to do some experimentation to see how well it actually functions :slight_smile:.

Edit:

For now, I have just stripped down the current PowerShell script to the minimum:

Add-Type -AssemblyName Microsoft.VisualBasic
if (Test-Path -LiteralPath ((Split-Path -Path $args) + "\~syncthing~" + (Split-Path -Path $args -Leaf) + ".tmp")) {
	[Microsoft.VisualBasic.FileIO.FileSystem]::DeleteFile($args,'OnlyErrorDialogs','DeletePermanently')
} else {
	[Microsoft.VisualBasic.FileIO.FileSystem]::DeleteFile($args,'OnlyErrorDialogs','SendToRecycleBin')
}

and also added -NoLogo to the command:

powershell.exe -NoLogo -ExecutionPolicy Bypass -File "C:\Users\User\Scripts\SendToRecycleBin.ps1" "%FOLDER_PATH%\%FILE_PATH%"

This has sped up processing quite significantly actually, or at least it feels like it. The current speed is ~330 items per minute, which equals to roughly 20,000 per hour, so it will take 10 hours to process 200,000 files. The problem is that this on a very fast machine, and everything becomes much, much slower on low-end hardware.

I also have a feeling that calling powershell.exe -Command "& {…}" instead of powershell.exe -File … may be faster, but I haven’t been able to make such a command deal properly with all the special characters that filenames can include. On the other hand, using a separate ps1 file has no problems with them.