Doing a test with a large folder of books, and when new books are added to the folder and subfolders, the main folder is renamed so we know the date of the current library.
Now this could be for anything really, but when the main folder is renamed, it looks like Syncthing treats that has a delete and create of the files, so the whole structure is re-synced again. Now when the size of the main folder is around 10G even over the wifi it can take a while.
Maybe when doing the scanning of the new folder, checking the hashs they could be compared to what is already known, to see if all that has to happen is the file needs to be moved not recopied over the network.
It does detect renames, so there must be something else involved which breaks that.
You can run with STTRACE=model, which will give you a more detailed explanation of what’s happening.
Ok. With just one of the machines connected (server) I have run the syncthing with the option provided.
Now on the command line on the folder doing a mv source dest
Did the rename and got flooded with messages like.
[67ZEU] 12:06:38 INFO: Puller (folder “Books”, file “Library2015.3.28a/Susan’s Books/Momotaro.pdf”): pull: no available source device
Where Library2015.3.28a is the old folder. Renamed it to Library2015.3.28b. This is version 0.11 under Debian Linux.
Then gets this
[67ZEU] 12:08:17 INFO: Puller (folder “Books”, file “Library2015.3.28a/Fiction2015.3.28/R/Robin Hobb/Golden fool - Robin Hobb.epub”): pull: no available source device
[67ZEU] 12:08:17 WARNING: Folder “Books” isn’t making progress - check logs for possible root cause. Pausing puller for 1m0s.
I’ll leave it to finish the scanning, and will check back soon… approx 10,000 files to scan and 8G
Here is some more now that it has finished scanning…
[67ZEU] 12:22:38 INFO: Puller (folder “Books”, dir “Library2015.3.28a/Fiction2015.3.28/D”): delete: remove /Users/fran/Books/Library2015.3.28a/Fiction2015.3.28/D: directory not empty
Also I notice that while it is scanning the number of files and size goes up.
I’m re-running it again as I had a client on my Mac that was still running in the before run, and this time I have renamed from b to c. This time when there is only the master on line, there are no messages on STDOUT.
Starting up the client for the 6.8G and 8649 files the following happens
[67ZEU] 12:46:40 INFO: Puller (folder “Books”, dir “Library2015.3.28b/Fiction2015.3.28/R/R Karl Largent/The Witch of SixKill (multi-html) - R Karl Largent/files”): delete: remove /Users/fran/Books/Library2015.3.28b/Fiction2015.3.28/R/R Karl Largent/The Witch of SixKill (multi-html) - R Karl Largent/files: directory not empty
[67ZEU] 12:46:40 INFO: Puller (folder “Books”, dir “Library2015.3.28b/Fiction2015.3.28/R/R Karl Largent/The Witch of SixKill (multi-html) - R Karl Largent”): delete: remove /Users/fran/Books/Library2015.3.28b/Fiction2015.3.28/R/R Karl Largent/The Witch of SixKill (multi-html) - R Karl Largent: directory not empty
[67ZEU] 12:46:40 INFO: Puller (folder “Books”, dir “Library2015.3.28b/Fiction2015.3.28/R/R Karl Largent”): delete: remove /Users/fran/Books/Library2015.3.28b/Fiction2015.3.28/R/R Karl Largent: directory not empty
and this is from the web interface on the client
So you can see that it wants to re-sync all of the files.
Not sure if it would help this case, cause I don’t get the errors the poststarted sees, but I took the liberty to do a similar test, but then with 10GB of Pictures (JPG and RAW). On both sides I started syncthing with STTRACE=model and renamed a directory on one node (node A).
What I do see is that node A and B syncs their databases but on node B I see it copying ALL the files into the .stversions directory as if the files where deleted/changed, then it takes a rename shortcut for all the files (from old directory to the new one) and finally it deletes all the pictures in the original directory (including the directory).
So not much network traffic here, but when it’s capable of seeing the rename, a copy off all the contents is just a waste of time and diskspace (unless you turn off versioning). Shouldn’t it detect first if it’s a rename before copying? If it’s a rename than don’t bother copying it in the .stversions directory.
edit: What version are you using Fran? I am using the latest v0.11.1 on Ubuntu 15.04 and Kubuntu 15.04
I don’t know why this occurs, but I had the same issue.
I fixed by completely removing the folder from Syncthing on all devices. Then I went to the actual folders on the devices and deleted:
.stfolder
.stignore
.stversions
I also have a custom hidden file that I created called .stignoreglobal so I deleted that as well.
Then I re-added the folder to Syncthing on one device, and shared it across to all the other devices. Then I finished by sharing the folder between all the other devices.
Again, I’m not sure why this worked but it fixed this issue for me.
Out of sync doesn’t mean it wants to resync all the files, it means that there is this much files to go through to work out where to get them from, which could mean locally too.
Also, the no sources available error is expected if you move files while they are being synced by someone else.
It seems that you don’t allow them to finish syncing the first before renaming, hence why they were transfered.
I have the same issue than Fran_Firman on synology arm V5 dsm 5.2 syncthing 12.15 and on another synology x64 dsm 5.2 syncthing 12.15.
When you rename a file an other one is created on the other side and all the files inside are downloded again! And the content of the old file is deleted.
excuse me for my english i’m french.
The first synology who syncthing installed, has a master record that is reflected in the second synology folder that has the slave … and when I rename a folder on syno A the folder is created on syno B but the content is not just moved,but re-downloded again and the files who should rename whas deleted.
It’s not clear to me whether you are moving files across syncthing folders, or across OS folders, nor did you answer my question of how do you know that the files are being redownloaded?
I see the traffic and the time it’s take to rename juste 3 or four album,it’s much difficult to explain when you don’t talk english!(do you speak french?)
You can enable STTRACE=model environment variable, and look at the logs at what’s happening. I think it will show a bunch of messages about rename shortcuts.
