I’m a happy Syncthing user and since about 3 years I have Syncthing running without any issues, syncing several TB worth of data between 6 or 7 machines.
This will be a very specific issue regarding the versions feature in Syncthing which I’m using as a backup (setting the local folder to “send only” / remote folder “receive only”) and I’m trying to explain it as best as I can, it’s probably a bit lengthy.
The main files in my case are projects created by Logic Pro (a music DAW, macOS only). Logics project files have 2 different ways of storing files: as packages or as folders. I work exclusively in the folder version of this, so that’s what I’ll be referring to.
In the folder version the project files (*.logicx) are actually directories structured like this, I’ve added " (directory)" to the name to show directories:
Of course not all of these files change when hitting “Save” in the software.
The version feature in Syncthing recreates this directory structure in .stversions and inside there will be timestamped versions of the files. So to go back to a previous version of the project I have to manually collect and rename these to recreate a valid .logicx project.
It would be great if syncthing would treat .logicx directories as 1 and create timestamped versions of this whole directory instead of timestamped versions of each individual file.
I understand this is a very specific exception and I’m just putting it out there, perhaps someone else runs into a similar issue with other software.
At this moment, the only way to achieve what you want would be to use External Versioning (see https://docs.syncthing.net/users/versioning.html#external-file-versioning) with your own script. Of course, you do need to have enough coding skills to write it first (or ask someone else to do it).
This is a great pointer and it sounds like a nice project that I may actually handle myself.
I can already anticipate how the script will have to travers up the directory tree for the files that Syncthing passes to me in order to do what I want, but I’ll find out once I see what Synthing passes to the script specifically and then deal with it.
Thank you! This sounds like a great project for me.
External Versioning is very powerful, yet it still operates on files rather than folders in principal, so you can’t version the whole folder, as on each run of the script you only know the path of a file that has just been modified or deleted (or renamed), however you can also you use the same script to manipulate the structure of the versions folder (which you’ve got no control of when using the other file versioning types).
I haven’t started writing the script yet, so I don’t really know what ST will present me with.
My naive thinking is that if ST reports that the file “ProjectData” was changed, I would just copy the existing .logicx directory two levels above with an added time string to the name to a different directory outside the .stfolder and then let ST just sync the new data coming in.
I guess I’ll find out when actually writing… I won’t be able to start until 12 days from now, but I am so looking forward to digging into this.
Ok, so today I started dealing with this and I have it almost solved.
However, one challenge remains:
So my task is, that when a file has changed within the .logicx project folder, I want to archive that whole project folder the way it was before the sync process started.
This I have solved, as soon as the first sync signal comes in, the external versioning system triggers my script and just moves a timestamped version of the .logicx folder to a separate directory.
But of course then ST gets confused because there are more files in the sync cue that should be replaced but suddenly ST can’t find the local files anymore and throws the “Local Additions” error. But of course all those local files have a size of 0 bytes, I assume because they no longer exist at that location.
One solution I have found when reading about this is, that I can of course create a cron job that triggers the “Revert Local Changes” command via curl. For some reason I don’t find that particularly elegant.
Is there another way of dealing with this situation?
Here is my script:
#!/bin/zsh
# Once the ST external versioning system triggers this script it will
# move the whole .logicx folder to an archive folder outside of the
# Syncthing folder
# Where I want my versions stored
versionspath=~/Desktop/Logic-Project-Archive
# The parameters we get from Syncthing
folderpath="$1"
filepath="$2"
# Test if the incoming file is a part of a .logicx path:
if [[ $filepath == *".logicx"* ]]; then
filetomove="$folderpath/""${filepath%%.logicx*}.logicx"
movedfile="$versionspath/$(basename "$filetomove" .logicx)-$(date "+%Y-%m-%d-%H%M%S").logicx"
mv "$filetomove" "$movedfile"
fi
You’re breaking the versioner contract. The versioner script is supposed to move the mentioned file away, in preparation for committing the new version of that file. It shouldn’t make other changes or you will indeed break the sync. In your case it looks to Syncthing as if you, the user, just deleted the directory while it was syncing changes to it.
You can make a copy of the entire parent directory to somewhere else if you want, since that doesn’t change the files being synced.
Yes, I know I’m breaking it, I’m trying to work it so it handles this specific data structure that Logic creates in a way where I can have a receive folder but keep functionally correct (!) project files for Logic.
If I use the normal file-versioning the Logic projects break and I have to manually re-assemble them (finding re-naming multiple files etc…). That takes too long as is too much manual work.
If I create copies then I will have multiple copies with timestamps a few seconds apart for every single time I save a project file on my local machine and again, they will not work correctly, because they all contain the wrong versions of the .logicx data structure (the first copy may actually be correct, but copy 2-4 will be wrong as they contain the first file from the current sync but file 2-4 will be what was there before the sync was triggered)
Yeah, maybe only do a copy if the previous copy is at least x time units old or something like that. But removing/changing other files from within the versioner is certainly going to cause chaos.
The main reason why I have to do it is, that the .logicx format is not a file but a folder that contains files - not all of which change.
So by default the versioner creates the files with timestampes, but that means that Logic can’t read those anymore. The versioner should create the folders with timestamps and the files must not be renamed. But as this is such a specific requirement there is no point in adding it natively to ST.
So I guess running the “revert local changes” curl script as a cronjob or a daemon is my only option.
I’m running the test right now and this does indeed do what I want.
The reason why I want ST to handle it is that it works so nicely and reliably in the background and I don’t have to worry about it. Carbon Copy Cloner could to the same thing on a timer, but more often than I like it looses connection to the server and I won’t know until I look at it.
This whole thing is not a real backup of my work. This is more or less a fail safe during the day in case I do some really stupid edit and have to go back to an earlier save. The real backups are to 2 alternating harddrives that are then backed up to Backblaze.
PS: I think the chaos is on the minimum side as the folder is receive only and thus just picks up the missing files again from the other machine after the curl command is issued.
So after a couple of days of observing this, I have to admit that I should have just listened to @calmh and acknowledged that ST is not the tool for what I want to do.
Calling the “Revert Local Changes” from the command line periodically then lead to the folder being stuck completely (I forgot the message now, but something like “Preparing folder sync” or something…).
I have abandoned the use of ST for this backup purpose and went over to an automated rsync LaunchAgent that does what I want.
I would have loved for ST to do this as it runs immediately after I save a project whereas my script now runs at dedicated intervals and thus takes a couple of minutes until the changes propagate, plus it has a huge overhead, because it has to scan the folders completely, but hey, whatever. It is what it is.
