Syncthing retransmits large files on rename

According to the FAQ and this:

Syncthing should detect file renaming and not retransmitting it. I am organizing a large collection os lectures (1GB to 3 GB) each file and renaming is part of the process. I am not talking about copying before deleting (without transmitting) but retransmiti ng files after renaming.

Please, remove the statement from the FAQ until (if not a feature) this is fixed.

-rsd

I don’t get what you want.

Syncthing should not retransmit renamed files.

From the FAQ: https://docs.syncthing.net/users/faq.html

Syncthing handles renaming files and updating their metadata in an efficient manner. This means that renaming a large file will not cause a retransmission of that file.

This is still true in most cases, yet subjective to how many files we are talking about per rename operation, which is probably worth while mentioning in FAQ. Also, if the files live in a different syncthing folder, this is somewhat expected.

Also, syncthing does not have a good way to indicate that the optimisation was taken, so you’d have to tell us how you verified that.

I am sorry I wasnt too clear.

They are in the same Sync folder:

$ find | wc -l 
450 
$ du -sh
322G    . 

All files were 100% synced between 2 hosts. I started to rename the files maybe 5~10 files per minute. Rescan is set to 10m. After a while, the out of sync files in the other host went up to 45GB. The upstrem went up (max). Looking at the out of sync files (pop up list of files), many files were showing 0B, so I assume it did detected the renaming procedure. OTOH, most files were showing the size (2GB, 3GB, …) which reflected the upload rate.

Stuff will still go out of sync and show large sizes of out of sync, regardless it will be renamed or not. The only way to know what exactly is happening is to enable debug logging.

If inotify is enabled, a few files a minute should be ok as it delays deletes compared to new files appearing.

If its on and you still have seen issues, it might be too much changes in a single scan period as the messages indicating change are capped in size, which the splits additional and removal of file over separate messages in which case the optimisation does not work.

It’s much worse when you rename directories, as then things are definately split over multiple messages.

If you don’t use inotify, then the bets are somewhat off for large files due to the message splitting business I’ve explained above.

As usual, the answer is not binary, but I suspect if you rename a single file, it should work as expected.

1 Like

However even that should result in copying the files locally, and later deleting them, instead of a rename, but not syncing actual data from a remote device, as we always send additions/changes before deletions. This should only break down if running out of disk space, in which case no new files are created, but deletions go ahead to free up space.

So we’d probably need model debug logs (probably also db for incoming indexes) to see what’s going on in your case.

1 Like

I will enable the debugs (model + db), wait for the remaining sync to finish and start a new batch of renaming to try to catch on logs.

Do you think that it would help:

  1. change the scan to a very large interval,
  2. disable inotify (watch for file changes),
  3. hit Rescan,
  4. rename the files
  5. rescan again

to get an usefull log?

I don’t have any clue how this problem is triggered, so from my perspective do whatever you need to repeat it - for the logs it doesn’t matter.

The more stuff happens in a single scan cycle, the more likely you’ll hit this problem.

You should keep inotify enabled as it makes the problem worse (by triggering smaller more often scans as it detects stuff).

Can you give me a (contrived) outline of how a rename can result in the deletion ending up sent before the creation? I fail to think of any (unless inotify messes up and only detects the creation long after the deletion, but that would strike me as very odd).

In inotify case i lt should never happen, as deletes are delayed, so worst case you end up with a copy (if blockmap still works, which we don’t know). In manual scan cases it can still happen.

Just for information, my system is an Ubuntu 19.10 with 32GB. The other host is a MacOS (don’t know the specs). I do have very long file names, up to 209 UTF-8 characters with accents (which makes them longer bytewise).

So, how can I send you the logs privately (as it has sensitive information to let it be posted publicly)?