Conflict files after network outage

Feature
1

Syncthing version: syncthing v1.2.2 “Fermium Flea” (go1.12.9 linux-amd64) deb@build.syncthing.net 2019-08-15 13:51:09 UTC

I found out that if two nodes are performing a change locally on there files during a network outage no sync files are getting generated.

Simulation of network issue: I crated iptables rule on node2 to disallow inbound/outbound connections for syncthing at 13:27 and flushed the rules at 13:48 During that time, I changed the same file on both nodes with different values

node1 logs: [NODE1] 13:27:49.939797 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/model/model.go:1434: INFO: Connection to NODE2XX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX at 192.168.0.197:22000-192.168.0.198:33302/tcp-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 closed: read timeout
[NODE1] 13:48:57.145423 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/connections/service.go:318: INFO: Established secure connection to NODE2XX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX at 192.168.0.199:54623-192.168.0.197:22000/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 [NODE1] 13:48:57.146522 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/model/model.go:1864: INFO: Device NODE2XX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX client is “syncthing v1.2.2” named “node2” at 192.168.0.199:54623-192.168.0.197:22000/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

node2 logs: [NODE2] 13:27:49.948873 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/model/model.go:1434: INFO: Connection to NODE1XX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX at 192.168.0.198:33302-192.168.0.197:22000/tcp-client/TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 closed: read timeout [NODE2] 13:48:57.158603 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/connections/service.go:318: INFO: Established secure connection to NODE1XX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX at 192.168.0.197:22000-192.168.0.199:54623/tcp-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 [NODE2] 13:48:57.158670 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/model/model.go:1864: INFO: Device NODE1XX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX client is “syncthing v1.2.2” named “node1” at 192.168.0.197:22000-192.168.0.199:54623/tcp-server/TLS1.2-TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

Nothing in the logs mentioned after the re-joins that a file was change or better a conflict was created/detected (no conflict file was generated).

Changes at the same time on both nodes: node1 logs: [NODE1] 13:58:13.741390 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/model/folder_sendrecv.go:1775: INFO: Puller (folder “mysyncthingtestfolder” (mysyncthingtestfolder), item “blafu”): handling file: file modified but not rescanned; will try again later [NODE1] 13:58:13.774286 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/model/folder_sendrecv.go:1775: INFO: Puller (folder “mysyncthingtestfolder” (mysyncthingtestfolder), item “blafu”): handling file: file modified but not rescanned; will try again later [NODE1] 13:58:13.799102 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/model/folder_sendrecv.go:1775: INFO: Puller (folder “mysyncthingtestfolder” (mysyncthingtestfolder), item “blafu”): handling file: file modified but not rescanned; will try again later [NODE1] 13:58:13.800025 /opt/tcagent/syncthing-1-work/174e136266f8a219/lib/model/folder.go:142: INFO: Folder “mysyncthingtestfolder” (mysyncthingtestfolder) isn’t making sync progress - retrying in 1m0.081608811s.

node2 logs: no logs generated

In this case, syncthing created as expected the conflict file.

Maybe this is something which would be good to have, because a network outage can happen.

2

Conflict files have no dependency on network connectivity.

Try to reproduce it again, and if you can, write up a detailed step by step guide (not random blob of text) how to reproduce it.

3

Hi, i tried it already some times and its always the same.

For reproducing:

  1. Disallow the connectivity between the nodes over the network (e.g. I did it with iptables on node2) ssh node2 > become root and placed iptables rules iptables -A INPUT -s -j DROP iptables -A OUTPUT -d -j DROP

  2. After some minutes you will see in the syncthing log that the connections are timeing out

  3. connect (ssh) to both nodes and change the same file but with differnt conntent

  4. wait some minutes (I waited for 20min)

  5. allow again the networc conectivity between the nodes (e.g. I just flushed the iptable ruleset with iptables -F) and stay connected to the nodes

  6. after the next sync happens the files are getting synced but without generating a conflict file

4

I presume you rescanned the folders after the modification?

5

The automatically rescann did that, so no other steps were taken as writen above.

6

Cannot reproduce, conflicts are created as expected (on 1.3.0-rc.2, but nothing changed recently regarding conflicts as far as I am aware).

7

What you describe here is the exact reason that Syncthing has the feature of “conflict files” - a file that has been changed on more than one node between syncs. Syncthing cannot know which is the “correct” change, and must present to the user all conflicted files so that they can make a decision on how to manage the conflict.

8

What does this mean? Does one file clobber the other? Do both files end up on both sides?

9

Hi, yea I know what it is but I can see that no conflict file is getting generated. One of the two modified files will be chosen and synced to the other node. The second file will disapear.

If you want to I can create you a log with STTRACE=sync,fs,event maybe that helps a bit?

1 Like
10

Hi, yea as written in my last comment, one file will be applied on both nodes and the second file just disapears and no additional conflict file is getting generated

11

Whoops, sorry for not reading your post properly. Is the case of both files the same? Has the case of either file changed at all? This can cause the “file changed but not rescanned” warning.

12

If you generate logs, model and db are most important, maybe scanner too. It will be terribly verbuous, but if you can reproduce you can make sure there’s no other activity at the same time, then that will help.

In general as you give clear steps to reproduce (thanks!) and I can’t reproduce, most likely there’s something peculiar (read different from mine) about your setup. So info on that might also give some clues (I tested on plain linux/ext4).

13

Ok, than I will generate the logs with that traces. As you expect a lot of logs, do we have any place in this forum where we can sotre it or shell I just attach it in a new post?

14

I don’t expect it to be that big - I’d probably upload it at https://gist.github.com/ or you can probably also attach it to a post.

15

Hello, I am super sorry that I did not responded for that long time. I don’t see the issue any more on our systems.

But thank you guys for taking time in this!