A possibly simpler alternative could be to have a receive-side configurable delay before processing deletes. That is, deletes are ”noted” but not performed until after a few minutes, during which time the files are eligible to be renamed instead.
1 Like
Yeah, I thought of that, and it would perhaps be simpler, but it would create user confusion, as it would not reflect quick-changes as quickly as possible.
And also, how little is little? What if I move 500 GB of data? How long would Bob have to wait for the scan of Alice to finish scanning? Would the deleted files stay in Bob’s folder as if nothing has happened till then?
Finally, why having to configure one more thing, and possibly make it brittle?
Moving them to .stcache solves all these issues, fair and square.
Basically my idea is not new, it’s re-hashing (pun) the .stversions hacky workaround already in practice, making it a standard practice.
There’s some root cause confusion here as renames should already be effective regardless of whether it’s 1GB or 500GB.
1 Like
I totally agree with that - it should be a consistent experience regardless of size.
I’m saying that as far as I know that is the case today. People keep saying renames need to be improved, but not showing a reproducible test case for when it fails.
Ah it failed with me just yesterday, still syncing! That’s why I came to this post!
I simply moved ~100 folders to a different subfolder (with 500GB/1000 files in total) and Bobs are receiving everything from scratch! It should have taken only the time Alice needs to rescan them + transmit hashes to Bobs, instead it’s being copying for 15 hours and still going!
Yeah… if you can describe a simple setup and steps where this happens reproducibly I’m interested, because this never happens for me.
ideally we want to be able to skip step 3,5, and 6..
Here’s another case.
-
enable versioning. Trash can is fine.
-
Create 10,000 files totalling 300GB in a subfolder…
-
wait for everything to sync
-
rename the subfolder.
what I expect is after some chugging/rehashing, that the effect of the renamed folder transfers to the remote device, with only data transfer of metadata, and I don’t have a mess of files clogging up the versions folder.
Nothing fancy.
I have a shared folder on 2 Windows 11 machines, with SyncThing 1.30 running, fully synced and not paused.
I have a folder with ~500GB with ~1000 files and ~300 folders. I select + move them via Total Commander to another folder in Alice.
Bob is receiving 500GB of data all over from scratch.
I’m not using the .stversions at all - I will have a try with that enabled and report back.
Feel free to try but versions “don’t exist” in the database and it doesn’t impact renaming. I have versioning enabled for other reasons and my expectation is renaming a folder doesn’t create a whole duplication of the renamed files in the versions folder. That’s why I stipulated it in my instructions.
1 Like
Does it have to be 500GB? You do this with 1GB and it works perfectly?
If so I wonder if it’s just that a “directory move” on Windows (or with Total Commander) isn’t like a directory move in Unixes. Over here, a directory move is just a rename of the directory, it’s effectively atomic. If this thing moves file by file and it takes a long time and it triggers a scan halfway through the operation, then yeah I could see weird stuff happening. If that’s the thing, you could see if increasing the watcher scan delay to cover the entire move operation works.
A partial sync while files are still being moved would first notice and announce the newly added files though, right? Why would it send the delete first without the new addition for even a single one of the moved files? Certainly the whole directory will be deleted last under its old name.
If it’s a long running file-by-file move we can start a scan, hash a bunch of the new files and discover they are renames, then we’ll scan for deletes (as we do after each scan) and see a bunch of deletes that happened during our hashing and thus won’t map to a rename. So that makes sense.
1 Like
Based on other replies to this, I see a need for a bit of clarification. Do you select the directory and move it, or do you select the content of the directory and move it?
1 Like
Directory renames on NTFS should just be a change in the MFT. Nothing else should happen. Directory moves on the same volume work the same way.
This is 100% the biggest pain point for us, but we have a strange use case, Avid Media Composer, which requires folders to be renamed in quick succession (to add media, then index it, then store it). Renaming a large directory… the scan starting… then renaming it again whilst that scan is ongoing will almost always cause the peers to immediately delete the whole directory and start again from scratch.
Why does this matter?
When a rescan occurs does syncthing have any knowledge that the newly named folder was born into existence via an atomic rename of an exciting folder?
Doesn’t it blindly scan folders and say, oh here’s a new folder with 1000 files, and oh that old folder with 1000 files is gone.
how does syncthing know if I duplicated everything to a new folder and deleted the old folder vs atomically renamed a folder?
Also FYI my renames are happening generally on a windows system and being synced to a Linux system.
1 Like
Admittedly, I was assuming that Syncthing was paying attention to the files themselves more than their filenames for this purpose.
In other words — and I do not claim to be expert on any filesystem — renaming a file or directory should result in the MFT being appropriately updated but not any bit of the file being changed.
The point is Syncthing is likely unaware of the MFT change and would likely scan the new file and could only detect that the hash was the same to assume it was actually a rename or move.