So I had a case where I processed 250GB of photos this week. Transfers from the camera, moves from one directory to another directory (within the same folder), file renames, and some more file moves (image sorting).
Normally while I’m doing all the moving and renaming I pause the syncthing folders. Then when I’m done, I pause the devices, and unpause the folders and let syncthing rehash all the files.
This time I decided to leave syncthing alone, and let it do its think while I did my thing (which is obviously a much easier way to work if I don’t have to worry about pausing and resuming.)
Much do my dismay, given a total of 250GB of files, I now have multiple copies with different file names spread across multiple directories in the versions folder. Total versions folder size is 1.5TB. (On average 6 copies of each file.)
So yeah. This can be improved.
Additionally, since I have now created a “folder” for the versions directory to try to save some of the data transfer, I find if I don’t pause the process to let the versions folder scan and calculate hashes, many of the files are transferred anyway.
There are a number of possible solutions. Perhaps none of which are really “easy”. In my mind, two solutions come to mind… Perhaps none are easy to implement. They both have their own pros and cons:
FIRST OPTION (optionally) set a folder to scan completely before sending info to remote folders. Let the hashing process complete and transmit all changes at once. On the receiving side, all changes are received and “processed”. Matches/renames detected, and handled before any other files needing to be requested are requested. In this way, renames result in the new file with hashes being sent in the same batch as the file delete (with the same hashes) and the renaming algorithm has a chance to do a rename as opposed to having the renamed file deleted first (added to versions), and then having to be redownloaded.
This is perhaps the best option… This I would think would be a folder specific option as many folders you may want the transfers to start ASAP, before scanning is complete, as happens today.
SECOND OPTION: Syncthing (optionally) indexed “versions” folders, and when a file is “deleted”, the file is moved to the “versions” folder, but didn’t have to be rescanned to be picked up. I.e. moved in the database also to the versions folder. Syncthing, if needing a file for which a 100% matching file is present in the versions folder, could move the file out of versions folder back into the proper location (and also renaming it).
If this were possible, my files could have been moved out and back into the directory and renamed as the changes occurred.
Anyway, I’m not saying I expect someone to be able to pick up these suggestions and make pull requests to address them. But boy it would be nice for myself and others that rename large files if this could be handled…