identical files 'out of sync'

Support
1

I have one node A for which 2 out of 5 other nodes see 2 or 3 files ‘out of sync’. On A, it shows it is synced fine with all other nodes. Looking at the incriminated files they are perfectly synced with the same ext4 permissions.

I restarted, looked at the model TRACE log on the boxes showing out of sync, tryied with ‘ignored’ permissions, looked at previous posts and can’t recognize my situation and sort it out.

Would this remind you a case you have seen before?

0.14.43 with Arch Linux or Win10 on the nodes

2

Even if the files are the same, in terms of content, permissions, mtime, syncthing still needs to reconcile internal differences, such as who is the device that saw the last change, which in case of two devices discovering the same identical file will not match.

Is one of the devices a master? Do you have ignore patterns that can cause this?

3

thanks very much for the response, there is no ignore patterns and no master among the devices

4

Can you post screenshot from both sides expanding the folder in question?

5

Do you need something more specific?

from acer, it is synced with VPC Screenshot_2018-01-30_13-36-49

VPC sees acer out of sync Screenshot_2018-01-30_13-35-07

the directory with the issue Screenshot_2018-01-30_13-44-10

VPC sees these files out of sync

Screenshot_2018-01-30_13-46-58
Screenshot_2018-01-30_13-46-58.png863×429 6.7 KB

6

also this is what the log shows on of the nodes that shows ‘out of sync’ for the incriminated ‘bin’ directory on acer

model.go:649: DEBUG: model@0xc42020a360 Completion(2BSTH72-ASYIO2K-UEH3ION-1234567-3QSKK2R-X6ICZYE-ET4XVUD-RY2THAY, "xwqea-39trv"): 80.082423 (4978 / 24993 = 0.199176)
7

You should post the view of the folder from both sides.

8

some more information about the files diverging

[6JY3H] 2018/01/30 14:14:22.693286 leveldb_dbinstance.go:405: DEBUG: need folder="xwqea-39trv" device=2BSTH72 name="bakcloud.sh" need=true have=true invalid=false haveV={[{PKASQZT 1} {5YQUZIY 2} {6JY3HKE 25}]} globalV={[{PKASQZT 1} {5YQUZIY 2} {6JY3HKE 65}]} globalDev=BSCETN2

[6JY3H] 2018/01/30 14:14:22.693615 leveldb_dbinstance.go:405: DEBUG: need folder="xwqea-39trv" device=2BSTH72 name="opennic.sh" need=true have=true invalid=false haveV={[{6JY3HKE 91}]} globalV={[{6JY3HKE 101}]} globalDev=BSCETN2
10

The 2 folder views on each side:

Screenshot_2018-01-30_14-42-30Screenshot_2018-01-30_14-42-52

11

Assuming your last logs are from VPC and 2BSTH72 (first block of id) is acer, then from VPCs perspective acer has {PKASQZT 1} {5YQUZIY 2} {6JY3HKE 25} for file bakcloud.sh, which is quite a bit out of sync with the global {PKASQZT 1} {5YQUZIY 2} {6JY3HKE 65}. To see what acer thinks about this file, could you please post the output of curl -X GET -H "X-API-Key: yourapikey" 'http://localhost:8384/rest/db/file?folder=xwqea-39trv&file=bakcloud.sh' | json_pp (adding your actual api key available in settings or config.xml). Also looking up which device 6JY3HKE is may provide a clue.

12

This is on acer (2BSTH72)

{
   "local" : {
      "version" : [
         "PKASQZT:1",
         "5YQUZIY:2",
         "6JY3HKE:65"
      ],
      "type" : 0,
      "size" : 1923,
      "numBlocks" : 1,
      "noPermissions" : false,
      "invalid" : false,
      "modifiedBy" : "6JY3HKE",
      "sequence" : 982,
      "name" : "bakcloud.sh",
      "modified" : "2018-01-28T19:43:06.179320483+01:00",
      "permissions" : "0755",
      "deleted" : false
   },
   "availability" : [
      {
         "fromTemporary" : false,
         "id" : "UBSOATL-"
      },
      {
         "id" : "6JY3HKE-",
         "fromTemporary" : false
      },
      {
         "fromTemporary" : false,
         "id" : "BSCETN2-"
      },
      {
         "id" : "5YQUZIY-",
         "fromTemporary" : false
      }
   ],
   "global" : {
      "deleted" : false,
      "permissions" : "0755",
      "modified" : "2018-01-28T19:43:06.179320483+01:00",
      "sequence" : 226,
      "name" : "bakcloud.sh",
      "modifiedBy" : "6JY3HKE",
      "invalid" : false,
      "noPermissions" : false,
      "numBlocks" : 1,
      "size" : 1923,
      "type" : 0,
      "version" : [
         "PKASQZT:1",
         "5YQUZIY:2",
         "6JY3HKE:65"
      ]
   }
}
  • It shows that from acer point of view it has the “6JY3HKE:65” while VPC thinks that it has “6JY3HKE:25”.
  • 6JY3HKE is the last device that modified the file and it is correct.
  • From the point of view of 6JY3HKE, I checked acer is also behind with version “6JY3HKE:25”.
13

Which device is this from? Can you provide the same from the other device?

It seems to me that one of your devices travelled back in time in terms of what the database looks like, as

[6JY3H] 2018/01/30 14:14:22.693286 leveldb_dbinstance.go:405: DEBUG: need folder="xwqea-39trv" device=2BSTH72 name="bakcloud.sh" need=true have=true invalid=false haveV={[{PKASQZT 1} {5YQUZIY 2} {6JY3HKE 25}]} globalV={[{PKASQZT 1} {5YQUZIY 2} {6JY3HKE 65}]} globalDev=BSCETN2

this says “I 6JY3HKE, have version 25, where as others have version 65 of the same file, that I myself have advertised”

14

This is on pc (6JY3HKE)

{
   "local" : {
      "deleted" : false,
      "modified" : "2018-01-28T18:43:06.179320483Z",
      "name" : "bakcloud.sh",
      "type" : 0,
      "size" : 1923,
      "permissions" : "0755",
      "numBlocks" : 1,
      "noPermissions" : false,
      "sequence" : 141,
      "modifiedBy" : "6JY3HKE",
      "invalid" : false,
      "version" : [
         "PKASQZT:1",
         "5YQUZIY:2",
         "6JY3HKE:65"
      ]
   },
   "global" : {
      "type" : 0,
      "name" : "bakcloud.sh",
      "size" : 1923,
      "deleted" : false,
      "modified" : "2018-01-28T18:43:06.179320483Z",
      "invalid" : false,
      "version" : [
         "PKASQZT:1",
         "5YQUZIY:2",
         "6JY3HKE:65"
      ],
      "permissions" : "0755",
      "noPermissions" : false,
      "numBlocks" : 1,
      "sequence" : 1131,
      "modifiedBy" : "6JY3HKE"
   },
   "availability" : [
      {
         "fromTemporary" : false,
         "id" : "BSCETN2-"
      },
      {
         "fromTemporary" : false,
         "id" : "UBSOATL-"
      },
      {
         "fromTemporary" : false,
         "id" : "5YQUZIY-"
      }
   ]
}
15

I confirm running the log again now on pc (6JY3HKE) it says

[6JY3H] 2018/01/30 15:29:33.236576 leveldb_dbinstance.go:405: DEBUG: need folder="xwqea-39trv" device=2BSTH72- name="bakcloud.sh" need=true have=true invalid=false haveV={[{PKASQZT 1} {5YQUZIY 2} {6JY3HKE 25}]} globalV={[{PKASQZT 1} {5YQUZIY 2} {6JY3HKE 65}]} globalDev=BSCETN2-

2BSTH72 is acer

16

And this file is causing an issue as well, this is the log I get from pc (6JY3HKE)

[6JY3H] 2018/01/30 15:34:01.928168 leveldb_dbinstance.go:405: DEBUG: need folder="xwqea-39trv" device=2BSTH72- name="opennic.sh" need=true have=true invalid=false haveV={[{6JY3HKE 91}]} globalV={[{6JY3HKE 101}]} globalDev=BSCETN2-

if I curl get this opennic.sh file on pc(6JY3HKE) I get

{
   "local" : {
      "type" : 0,
      "name" : "opennic.sh",
      "sequence" : 91,
      "numBlocks" : 1,
      "invalid" : false,
      "deleted" : false,
      "modifiedBy" : "6JY3HKE",
      "noPermissions" : false,
      "version" : [
         "6JY3HKE:101"
      ],
      "permissions" : "0755",
      "modified" : "2018-01-28T11:04:51.882622578Z",
      "size" : 3055
   },
   "global" : {
      "name" : "opennic.sh",
      "sequence" : 1128,
      "type" : 0,
      "noPermissions" : false,
      "invalid" : false,
      "numBlocks" : 1,
      "modifiedBy" : "6JY3HKE",
      "deleted" : false,
      "version" : [
         "6JY3HKE:101"
      ],
      "permissions" : "0755",
      "modified" : "2018-01-28T11:04:51.882622578Z",
      "size" : 3055
   },
   "availability" : [
      {
         "fromTemporary" : false,
         "id" : "BSCETN2-"
      },
      {
         "fromTemporary" : false,
         "id" : "UBSOATL-"
      },
      {
         "fromTemporary" : false,
         "id" : "5YQUZIY-"
      }
   ]
}

and the same opennic.sh file on acer

{
   "global" : {
      "numBlocks" : 1,
      "modifiedBy" : "6JY3HKE",
      "invalid" : false,
      "modified" : "2018-01-28T12:04:51.882622578+01:00",
      "type" : 0,
      "noPermissions" : false,
      "size" : 3055,
      "deleted" : false,
      "version" : [
         "6JY3HKE:101"
      ],
      "sequence" : 227,
      "name" : "opennic.sh",
      "permissions" : "0755"
   },
   "local" : {
      "modified" : "2018-01-28T12:04:51.882622578+01:00",
      "invalid" : false,
      "modifiedBy" : "6JY3HKE",
      "numBlocks" : 1,
      "permissions" : "0755",
      "version" : [
         "6JY3HKE:101"
      ],
      "sequence" : 978,
      "name" : "opennic.sh",
      "deleted" : false,
      "noPermissions" : false,
      "type" : 0,
      "size" : 3055
   },
   "availability" : [
      {
         "id" : "UBSOATL-",
         "fromTemporary" : false
      },
      {
         "fromTemporary" : false,
         "id" : "6JY3HKE-"
      },
      {
         "fromTemporary" : false,
         "id" : "BSCETN2-"
      },
      {
         "id" : "5YQUZIY-",
         "fromTemporary" : false
      }
   ]
}
17

Did you have some sort of database rollback or something of that kind?

@calmh, ideas?

18

A database rollback? Not that I know… is there something I can do to reset things because there is something wrong with my db it seems indeed.

Talking about the file opennic.sh for example both pc and acer have version 101 and the global version is 101 looking at the curl get output. Then there is this version 91 in the log, where does it come from?

19

So there is. Shut syncthing down and run syncthing -reset-deltas, if that doesn’t help, do -reset-index, but in the meanwhile, save a copy of the db for inspection. Its path is printed in -paths.

If you can wait, it would be interesting to see what @calmh has to say, as we might want to do more debugging.

1 Like
20

-reset-deltas (only, no other action) solved the issue on pc and I saved a copy of the db for further debugging if needed. I am available to do some tests with it.

Thanks very much for your insight and your help.

21

when you say -reset-index do you mean -reset-database? I don’t see -reset-index documented here https://docs.syncthing.net/users/syncthing.html