I have observed that Syncthing spawns at most two PowerShell windows at a time, and then spawns new ones only once one of the two previous ones has exited. The problem with this is that a) PowerShell itself is rather slow to start, so it always takes some time to spawn the process, and b) even without considering the PowerShell’s speed, it is still slow to process a large number of files this way.
Is there any way to increase the number of these processes allowed to run at the same time? Now, I need to disclaim that I haven’t analysed the code responsible for the external versioner, but is the number of copiers perhaps involved in it in any way here? Would increasing copiers help the external versioner in spawning a larger number of processes at a given time?
I would prefer to stick to built-in solutions only, but regardless of the tool, Syncthing still spawns only two instances of the process anyway.
I’m going to sync around 200,000 deletions very soon, and I have temporarily increased the number of copiers for the folder in question, so we are going to see whether there is any difference or not.
I can now confirm that the number of copiers does not seem to matter. From what I can see observing the spawned processes, Syncthing seems to simply version the files one after another, without any actual multithreading.
On fast hardware/storage, this is not a huge deal, as still a couple hundreds of files are managed to get processed per minute, but on slower machines, it can take a long while to go through all the files.
I guess that it may only seem like it in the Task Manager, which probably has some kind of a refresh delay, while in reality Syncthing just keeps opening only one new window at a time.
I do understand what you mean, although it would probably be safe in this case, as Windows gives a unique identifier to each file that goes to the Recycle Bin. Anyhow, so I guess that I will have to think about how to make each process open and close as fast as possible to make the queue move faster…
I am not concerned about the safety from windows side.
I am concerned about something going bad syncthing side. Can’t think of anything off the top of my head, but I am not sure that it would all be good either.
First, call a one-line batch script just to move the file to a temporary folder on the same partition. This should execute instantly.
Then, schedule a PowerShell script to run every hour or so and move all the files from that temporary folder to the Recycle Bin in one shot.
This should be much faster, because cmd.exe has much smaller footprint than powershell.exe, and also calling PowerShell on a folder with multiple files only once will be much quicker too.
If you want it fancier, you can even run something constantly, listening on folder status events, instead of hourly, to move the Syncthing trash can to the windows trash can whenever Syncthing finishes syncing.
Yeah, although I would prefer to keep everything in a single folder (for the PowerShell script), and the default versioning will likely overwrite files with the same names coming from different Syncthing folders. Also, there is a different problem that I haven’t considered yet, i.e. recycling files from their non-original location will make restoring them later much more difficult (as everything will go back to the temporary folder instead of the original locations).
I probably need to think about this more in order to come with a decent solution that will be lighter on the resources and definitely quicker, as right now it may take hours or possibly days just to proceed all the versioned files.
This has sped up processing quite significantly actually, or at least it feels like it. The current speed is ~330 items per minute, which equals to roughly 20,000 per hour, so it will take 10 hours to process 200,000 files. The problem is that this on a very fast machine, and everything becomes much, much slower on low-end hardware.
I also have a feeling that calling powershell.exe -Command "& {…}" instead of powershell.exe -File … may be faster, but I haven’t been able to make such a command deal properly with all the special characters that filenames can include. On the other hand, using a separate ps1 file has no problems with them.
Forgive me to reply to this script without detailed knowledge on the subject or the required action.
What I see here, is a powershell sscript that does file actions using other scripts. Why not use the powershell native file actions? Why not use the VisualBasic script? or even better, direct .net as that’s what I think is used in the end. That’s just some general ideas to speed up.
Next thing: In the if-test, you use a file-system-access as far as I know only to see if the file name ends with ‘.tmp’. That could be changed to just use string handling stuff. That’s much faster.
In the end, why bother? just move all to the recylcebin (or delete it all).
[UPDATE] I peeked at the source and I see you use the example code to do your own file-versioning… and use the secundary example given. If you are asking the above type questions and are not capable of debugging or altering the given scripts, better use the file-versioning as provided by SyncThing itself. Those own versioning scripts are for engineers and technicians that do know what they want and are capable scripting that.
On the examples given: If I had the time, I’d either remove them or update them.
[update again]
Replace the realy delete code (the ‘then’ action) with the realy powershell action ‘Remove-Item’. See the powershell documenation on details to do it with the same force and verbose as in the script.
For the record, I am not using any versioning right now, because it was slowing down the actual synchronisation too much (mainly because I sync tonnes of small files that change often).
As for the rest, if you have a better approach to the recycling script, please feel free to update the Docs. The current one works fine for basic needs, e.g. for a typical documents folder or such. The problems arise when applying it to a folder with tens of thousands files that get changed and/or deleted often.
PowerShell has no native way to recycle files, hence the whole need for Visual Basic. Moving them physically to the Recycle Bin folder is not what Windows does. They need to be renamed and added to the index with all the required information (path, filename, date, etc.), so that they can be later listed and potentially restored, e.g. using the native Windows Explorer interface.
I have tried that. There is no way to get rid of the Windows Explorer file delete confirmation pop-up with that method.
.