Questions about the way synchronisation works

Hi I’m experimenting SyncThing because it seemed to be the solution I’m looking for. I have some remarks:

  • could we have a button to scan immediatly for differences, it would be useful for beginer’s understanding of the process.

  • Bug : when deleting a file which is in read only mode (node win 7 / node win 8), It won’t delete on the other node, so the file is always coming back.

  • I meet difficulties to understand the way suppressions work. Some times, I must supress the same directory more than once. Even some times, supressing from the othe node is the only way, because some empty directories always come back. Do you have a text explaining the problem of synchronisation?

  • I’m sure it would be clear in version 1 and above: is it possible to hide the Dos windows where runs syncthing.exe ? Well I have seen that some messages are prompting here, but none of them helped me in understanding of what happened while deleting things!

Look at https://github.com/calmh/syncthing/issues/ there are some of out points.

e.g.:

Deleting folders does not always work -> https://github.com/calmh/syncthing/issues/198

See:

Sounds like a bug, yes. Could you file an issue here? https://github.com/calmh/syncthing/issues

I’m not sure I understand the issue here. Are you talking about .stignore files?

See:

Thank’s for the answers. I have been wrong to post 4 questions on a single post. Now I must clear things.

  • Button

For the button problem, I just changed the setting of the rescan delay from 60 to 10 seconds and it’s fine that way.

  • Deletion bug

Done : Fails to delete read-only files #236

  • Sync sheme (the initial thread’s aim)

I’ll make another answer just after this one to try to clarify some things about the “unsuppressable” folders and for my comprehension of the sync process.

  • Hiding ST windows

It is not a real problem for the moment, just a bit annoying, and you will certainly confirm to me that there will be later a better level of encapsulation on different platforms.

Sync

I don’t post an issue, because I’m unable to reproduce the behaviour I will describe here.

  • Config ST v0.8.7, Windows (7 and 8), 2 Nodes, 1 repository.

  • Action I create a folder, wait for it to be synced, delete it from any node.

  • Problem The problem occurs some times: the folder in the othe node is’nt suppressed during the sync, and is recreated during the next sync where I deleted it.

  • Questions I imagine that, just like the problem of the ‘Read-only’ File attribute, if some problem occurs on the remote folder, ST will silently continue it’s work and that, on a second sync, the folder (and not it’s content) will be recreated where it was deleted. In this case, the problem is the silence. I even can imagine that there can be a synchronisation problem in a node’s process or even between the two node’s process. Sync error on one node: I did a lot of experiment and I dont’ remember all of them, but I seem to feel that maybe(!) the deletion order on the remote node could be wrong, letting the process try to delete the root folder before deleting all of it’s own components, producing a silent error as described before. This can’t explain the issue described firstly in this message with empty folders. Sync error on two nodes: I imagine that if two processes are active at the same time on the same deleted empty folder, maybe they can interpret differently the situation, only one of them being the winner. Which brings me to a big question:

How does work the sync for deleted or created files/folders? How does it decides wether the file must be deleted or sent to other nodes?

For instance:

· I turn of sync on both nodes; · add a new file1 in node1; · delete an existing file2 in node2; · turn on the sync on both nodes.

How does ST manages the sync?, how does it decide that:

· node1/file1 must be sent to node2/file1; · node1/file2 must be deleted ?

There is an “index cache” kept that contains the state of the local repository. When syncthing starts, it reads the index and compares to the current state of the repo. Files that are in the repo but not mentioned in the index are new. Files that are mentioned in the index but not present on disk have been deleted.

The process is basically the same for each rescan when running, except the index is already in memory and represents the state of the repo at the previous scan.

Files that we think we deleted but then pick up on the next scan will be seen as “new” again.

3 Likes

Explanation clear, thanks.

This must be the hot-point of the sync, which must be able to manage so many situations.

Example: what if one tries to sync a repo on a removable drive, that one moves from a computer to another, modifying the repo and eventually declaring it more than one time as a node on distinct computers? It is a rethorical question, I don’t specially expect you to answer it!

This is a very, very useful information, IMHO it should be collect into github wiki, discuss here: Collect documentation into github wiki

Question: What happens, if i restart syncthing while index cache creation?

You’ll confuse syncthing and it’ll eat all your data for breakfast, probably. :slight_smile:

It will use the previous cache file and restart the scan, or if it was the very first startup it will do the scan from the beginning again.

Sounds good. Then syncthing saved the information that a initial scan is completed?

My my fear was, that syncthing means all not scanned files are all missing files, so they are new.

No, that shouldn’t happen. It writes the new index to a temporary file, and when complete replaces the old file with the new. Likewise changes aren’t “published” to it’s peer nodes until the scan completes.

Other related question:

syncthing can’t known about file renaming/moving, isn’t it? If i rename file A to file B, then it can only collect the information “file A was deleted and file B was created”, the same with moving, isn’t it? Is it right that this can only be handled if inotify is used, or?

Correct, that’s how it’s currently handled. However, it could be handled nicer, inotify or not. We know the hashes of both new and old files, could see that they are the same and do the move efficiently. https://github.com/calmh/syncthing/issues/3 (old one :slight_smile:

Sorry to resurrect this thread, but my questions seemed to fit (hope I didn’t miss the answers along the way)

I just added a folder to a test repo with a git repository. Four of the folders under the .git directory were empty on node1 and they were not synced to node2. However when I manually added an empty folder to node1, it was added to node2. Is that expected behavior? (Both nodes 0.8.17 on Linux). Now on node1, 'ls /mnt/testrepo | wc -l = 2081KK.

I tried this on my home network which is significantly faster and this didn’t happen. Could it just be product of the slower network?

Thanks, Scott

Empty folders should be synced irrespective of whether they were created before or after initial sync nor the speed of the network, so that sounds a bit weird.