Files Unexpectedly In .stversions - Sync Chain?

Support
1

Hello:

I’m troubleshooting an installation that we’ve recently set up, where we’re unexpectedly finding some files in the .stversions folder.

According to the timestamps, they were moved in there over a month ago (only just noticed them) - so we’re not going to see them in the Recent Changes list. The user is adamant that the files hadn’t intentionally been deleted from the cluster and, whilst we’re glad the file versioner had caught them, we’re wondering why they went walkabout in the first place.

The sync cluster was set up with a folder full of files on one machine, and empty folders on three other machines ready to receive them. All folders were set up as Send/Receive type.

One thing that strikes me here is that the Syncthing instances were set up to only communicate with their immediate peers - so:

A <-> B <-Internet-> C <-> D

…instead of…

A <-> B

A <-> C

A <-> D

B <-> A

B <-> C

B <-> D etc.

Would you expect this to cause issues as changes propagate up and down the chain? Or is there anything else you might think might have caused this?

Thanks!

2

The only things that cause files to end up in versioning directory is modifications or deletes, topology has nothing todo with this.

3

Hi Audrius:

Ok - thanks, that’s reassuring on some levels. I still don’t know what’s caused the files to go off.

It’s as though, mid-sync, a recipient decided its (incomplete) copy of the folder was the most-recent version, causing the ‘master’ to then ditch its copies of the outstanding files.

I will continue to investigate…

4

Hello again:

Got a bit more information here. To be clear: looks like we don’t have any actual data loss - but this looks like a weird edge case that I presume we would like to see fixed!

Assuming topology isn’t an issue here, there are two machine relevant in the Syncthing cluster: a user-operated Mac (AOYVY6K) and a NAS on the local network (LUQDG7T). A folder full of data was dropped into a newly-configured shared folder (theoretically (and probably) empty in all locations) on the Mac, in order that it syncs up to the other devices in the cluster.

Most of the data is in place - but some files in some folders are now in .stversions on the Mac, and are completely missing on the NAS. Whilst this was going on, no-one was making any changes on the NAS - so there shouldn’t have been anything causing the files to get deleted.

Here’s the information the API (on the Mac) is reporting for a file which is correctly in place on both Mac and NAS:

curl -k -X GET -H "X-API-Key: ******" 'http://localhost:8384/rest/db/file?folder=*****-*****&file=path/to/file1'`
`{`
  "availability": [
    {
      "id": "LUQDG7T-******",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2010-07-06T10:44:31+01:00",
    "modifiedBy": "AOYVY6K",
    "mustRescan": false,
    "name": "path/to/file1",
    "noPermissions": true,
    "numBlocks": 1,
    "permissions": "0770",
    "sequence": 18949,
    "size": 111915,
    "type": 0,
    "version": [
      "AOYVY6K:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2010-07-06T10:44:31+01:00",
    "modifiedBy": "AOYVY6K",
    "mustRescan": false,
    "name": "path/to/file1",
    "noPermissions": true,
    "numBlocks": 1,
    "permissions": "0770",
    "sequence": 727,
    "size": 111915,
    "type": 0,
    "version": [
      "AOYVY6K:1"
    ]
  }
}

And here’s the information the API (on the Mac) is reporting for a file which is incorrectly in .stversions on the Mac, and missing on the NAS. This file should be in the same folder as the file detailed above:

curl -k -X GET -H "X-API-Key: ******" 'http://localhost:8384/rest/db/file?folder=*****-*****&file=path/to/file2'
{
  "availability": [
    {
      "id": "LUQDG7T-*****",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": true,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2018-12-02T20:43:00Z",
    "modifiedBy": "LUQDG7T",
    "mustRescan": false,
    "name": "path/to/file2",
    "noPermissions": false,
    "numBlocks": 0,
    "permissions": "0",
    "sequence": 7016,
    "size": 0,
    "type": 0,
    "version": [
      "AOYVY6K:1",
      "LUQDG7T:1"
    ]
  },
  "local": {
    "deleted": true,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2018-12-02T20:43:00Z",
    "modifiedBy": "LUQDG7T",
    "mustRescan": false,
    "name": "path/to/file2",
    "noPermissions": false,
    "numBlocks": 0,
    "permissions": "0",
    "sequence": 33453,
    "size": 0,
    "type": 0,
    "version": [
      "AOYVY6K:1",
      "LUQDG7T:1"
    ]
  }
}

Unfortunately, as this happened over a month ago, we have no recourse to logs or other forensics - unless there are other API probes you can suggest to narrow down what has happened here.

Any thoughts? Thanks!

5

Can you get the same data for the missing file on the other side?

6

Sure - here you are:

For the same present and correct file (‘7SYXIZ5’ is a NAS at a remote location):

curl -k -X GET -H "X-API-Key: *****" 'http://<NAS-IP-Here>:7070/rest/db/file?folder=*****-*****&file=path/to/file1'
{
  "availability": [
    {
      "id": "7SYXIZ5-******",
      "fromTemporary": false
    },
    {
      "id": "AOYVY6K-******",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2010-07-06T10:44:31+01:00",
    "modifiedBy": "AOYVY6K",
    "mustRescan": false,
    "name": "path/to/file1",
    "noPermissions": true,
    "numBlocks": 1,
    "permissions": "0770",
    "sequence": 11306,
    "size": 111915,
    "type": 0,
    "version": [
      "AOYVY6K:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2010-07-06T10:44:31+01:00",
    "modifiedBy": "AOYVY6K",
    "mustRescan": false,
    "name": "path/to/file1",
    "noPermissions": true,
    "numBlocks": 1,
    "permissions": "0770",
    "sequence": 18949,
    "size": 111915,
    "type": 0,
    "version": [
      "AOYVY6K:1"
    ]
  }
}

…And for the same missing file:

curl -k -X GET -H "X-API-Key: ******" 'http://<NAS-IP-Here>:7070/rest/db/file?folder=*****-*****&file=path/to/file2'
{
  "availability": [
    {
      "id": "AOYVY6K-******",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": true,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2018-12-02T20:43:00Z",
    "modifiedBy": "LUQDG7T",
    "mustRescan": false,
    "name": "path/to/file2",
    "noPermissions": false,
    "numBlocks": 0,
    "permissions": "0",
    "sequence": 33453,
    "size": 0,
    "type": 0,
    "version": [
      "AOYVY6K:1",
      "LUQDG7T:1"
    ]
  },
  "local": {
    "deleted": true,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2018-12-02T20:43:00Z",
    "modifiedBy": "LUQDG7T",
    "mustRescan": false,
    "name": "path/to/file2",
    "noPermissions": false,
    "numBlocks": 0,
    "permissions": "0",
    "sequence": 7016,
    "size": 0,
    "type": 0,
    "version": [
      "AOYVY6K:1",
      "LUQDG7T:1"
    ]
  }
}

Thanks!

7

@calmh @imsodin it looks like this was a conflict? Or versions being the same is expected? Reason I ask is because the other ok file has a single version.

8

I don’t see how it could be a conflict. The version of the delete file looks just how it is expected if the file were deleted on the NAS (A:1 for creation by mac, then update with L:1 for deletion by nas, which is the last modifying device). And nas has nothing, so what would it be conflicting with. I may be having a blockade, I don’t see any way how the describe scenario could happen (which does not mean I think that it didn’t happen :wink: ).

9

I thought it should be a:1 b:2 if b modification happened after a and b was aware of a’s modification?

10

A version update just searches for the id of the updater and increases the associated, id-specific counter by 1.

11

Ok, I don’t think syncthing did anything wrong.

I don’t know why your files disappeared, but it seems the NAS received them and after that they went missing from the local filesystem for some reason. Perhaps the NAS does not respect fsync and the machine was abruptly restarted, which essentially caused the files to disappear after being recorded in the db.

12

Hello:

Ok - I realised it would likely be difficult to track this without any log files, but thanks for giving it some thought anyway!

Just in case this is an easy one to spot (and I’m sure I’m working at a level faaaar below you guys!): do files get recorded in the database before they are confirmed written to disk? Or does the database only get updated after the file write is completed?

13

Yep, logs from NAS during the event (model, db or fs debug facility services) could shed more light. Mostly after the fact it is hard/impossible to find out what exactly happened.

Only after. And that’s where Audrius’ scenario comes in: If your filesystem reports the file was written successfully, but it wasn’t, on the next scan Syncthing will detect that there is no file on disk, so it marks the file as deleted in the database.

14

Ok - thanks!

I’m not sure what scenario would have caused files to not be permanently written to disk here -it’s a mature platform - Synology NAS on BTRFS, and it didn’t crash at the time. However, I appreciate it’s not going to be possible to further track down without logs. Oh well!

Thanks for your inputs anyway!

15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.