I tried to find the state of this and found those:
I am sure, there are more threads.
So, it seems that SyncThing can handle local renames of single files when there is a new (automatic or manual) Rescan after each such renaming, right?
When there are too many such renames, Syncthing messes up the different add and remove commands and things may not work as expected.
This makes renaming of many files and esp. directories … nearly impossible (without a new network transmission, that is).
So my main question here is, if there is something planned to make it possible to rename many files?
One implementation would be by stopping automatic scans (maybe with a button “start local renaming”), then applying all local renames and then start some “rename” scan (with a button “scan for renamed content”) so that all renamed files can be found and renamed on the remote site too?
That should not be hard to implement, just adding a new button that explicitly searches for renamed content - just like git does it, based on the two sets of hashes.
Would it work that way when you pause all remote devices, but let the folders keep scanning normally? Then after all renames have been scanned, resume the devices.
It’s not clear to me that the benefit of this feature would be worth the development cost, since there’s an easy way to automate a solution without digging into the way scanning works.
Note, however, that I don’t speak for the Syncthing team and I’ve never been a developer, so I could be full of baloney.
That means you have to start some script before you do a big rename and start another script (or pass through a pause point) after the renames are done?
Nah. That’s a terrible bandaid to a problem that many other syncing tools have already solved and based on the many many threads is a common expectation.
I know the implementation isn’t simple but there are was to do this. They have haven’t been implemented yet.
Perhaps. But if you want to rename a directory with 500GB of files spread over 1000 files you should be able to just rename the directory without triggering a massive resend…
Especially because in many (but not all cases) syncthing is getting notified of a directory rename via the watcher system.
Anyway don’t get the idea I don’t love the tool. It’s fantastic. But still. This could be improved.
From my perspective, you won’t add some lines of script when you’re already using one for a highly unusual corner case, denigrate the tool and our assistance and demand that a core function be rewritten.
I have no idea how this could be done.
Can you explain, please?
Syncthing, on the other hand, already has everything need for single and mass renames. It may just need a special mode for such operations so that the renames get detected correctly.
I doubt adding a “special mode” just for renames is desirable. On the other hand, I’m sure improvements to the current code in order to handle mass renames better would be very welcome.
I’ve personally solved this problem a while ago by using versioning while also adding all .stversions folders to Syncthing as unshared folders with fsWatcherDelayS set to 1. This way, even if rename detection has failed, Syncthing can still almost always re-use local files from the versioned folders instead of re-downloading them from scratch. I may write a quick tutorial on this method in a separate topic sometime in the future.
One bit that might help with your approach @tomasz86: The need to watch the .stversions folder is a bit wasteful. Syncthing could instead do an internal change detection in this specific case. Whenever a file is moved to the versions folder (or any other time when Syncthing really calls a move operation), it could check whether the move target is inside one of the shared folder root paths. If so, it should trigger a rescan of that path on the destination, avoiding the need for the watcher to notice the new file.
So I have implemented this approach when you suggested it in the past.
There are two main problems that arise when the directories or files renamed are large.
Consider a 300GB directory renamed.
if the rename isn’t picked up then I have 300GB of files moved to stversions. These files need to be hashed before they can be used as copies. It seems the transfers would start before the hashing has processed most of the files.
I now have two copies of all of these files. Which for 300GB is a substantial waste of space. I could set the stversions retention to something like 2 days but I actually want to retain files here as a primitive backup for about 6 months. (Don’t worry this is combined with a real backup strategy.)
I’m really not trying to be difficult. And while mass renaming of many large files may be an edge case to some extent it is simply an extreme case of a routine operation that pretty much everyone does. I mean don’t we al rename files or directories occasionally.
Yes I know for smaller files the retransmission can be quick and almost invisible. But those duplicate files do clutter the versions folder. (Whether you’re indexing it or not.) And they really don’t belong there.
“Hey I renamed this folder. The rename was synced with the remotes, but a copy with all the files and the old directory name is in the recycle bin.”
I posted some ideas on solutions or improvements a while back.
Thinking a bit more, if we are using the fswatcher then we know about the renames, does it make sense that when the watcher identifies a rename ST could scan the renamed file and confirm it’s the same as what’s in the index and if so, transmit a rename instruction? There wluld have to be some verification that everything is in sync and we don’t end up with conflict files being created if there’s some “confusion”. But it would seem like on the surface we should be able to keep stuff in sync if we transmitted what the watcher stated actually happened. Unless the watcher can’t be trusted.
We have an issue for that (see https://github.com/syncthing/syncthing/issues/5279) . I think one advantage of my approach is that you can use different fsWatcherDelayS per folder, and setting it to 1 just for .stversions makes Syncthing hash those files almost immediately. However, I don’t actually add all the .stversions folders separately though. As my Syncthing folders are all located in the same root path, I only add the root path itself with
!.stversions
/
for the ignore patterns. This makes things much simpler to configure and manage.
Yeah, there will be edge cases, although I’d argue that renaming such giant folders isn’t something that you do every day. In my case, the approach has been working fine so far with folders up to around 50GB or so. The hardware does matter though, as it likely won’t work as well on slower devices (although mine aren’t exactly top-tier either).
It is true that disk space may be an issue. In my case, I use simple versioning which is set to delete those temporary versions after just “1 day”. I’ve got separate devices that utilise staggered versioning set to much longer (or infinite) retention periods strictly for backup purposes.
Just for the record, the reason why I came up with this strategy in the first place was because one of my location runs on mobile data, which is limited, and I used to run out of it regularly. After starting reusing versioned files as described above, the problem has disappeared.
I still have no idea about how scripting could improve the situation and how exactly this would need to be done.
Whatever, I found this:
And this involves manually renaming files and folders on local and remote devices.
I cannot do this.
All remote devices are not mine and I have no access beside what Syncthing offers locally. So I can only pause the folder at the local side or the remote device itself.
Does this help in renaming?
How to handle this best?
For example, you could use a script to pause the folder first, then do all the renaming (manually), then use a script to unpause it. Basically same as using the Web GUI but faster.
I believe just pausing the folder locally and doing all the renaming while it stays paused should be enough for Syncthing to detect the renames properly and not re-upload the files after you unpause it.
. I think one advantage of my approach is that you can use different