Not syncing properly, database error

Support
1

I have the issue that Syncthing reports my folders in sync even though they are gravely out of sync. I suspected something might be wrongly ignored/incompletely indexed, so I queried the API and received an error:

❯ synct 'db/browse?folder=data&levels=3'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    96  100    96    0     0    387      0 --:--:-- --:--:-- --:--:--   387
   1 could not find child 'chores' for path '1-projects/falcon/chores/.gitignore' in parent 'falcon'

Where synct is a little helper to curl the API.

What is this error?

Note that the chores folder has been deleted on another machine, which I can also see in the index if I query only 2 levels deep.

2

Smells like Syncthing has issues with deleted/moved folders. I deleted the folder and the next error about a deleted folder popped up. How can such a basic functionality fail all the time?

3 
❯ synct 'db/browse?folder=data&levels=4'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   112  100   112    0     0    262      0 --:--:-- --:--:-- --:--:--   262
   1 could not find child '.github' for path '1-projects/software-challenge/docs/.github/workflows' in parent '
     docs'

deleted that folder, rescanned:

❯ synct 'db/browse?folder=data&levels=4'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   125  100   125    0     0    229      0 --:--:-- --:--:-- --:--:--   229
   1 could not find child 'lib' for path '1-projects/software-challenge/gradle-test/lib/build.gradle.kts' in pa
     rent 'gradle-test'

feels like a neverending story…

4

The issue apparently is that I need the (?d) prefix (Ignoring Files — Syncthing v1 documentation).

Nonetheless, the database shouldn’t error out without that.

5

That endpoint isn’t used for anything internal. You have found a usability issue in that, but that doesn’t affect syncing.

Looking at the code this error can occur when a parent directory of an existing file has been deleted on a remote. This can happen with non-identical ignore patterns. And it should be handled in code, i.e. the directory is re-created when scanning/syncing. If that’s not the case, please describe what the situation is for you (paths, screenshots) and we can have a look.

6

The ignore patterns are fully identical on both machines, but that issue isn’t as bad as what I currently have:

  • I deleted a folder (which was already ignored anyways) on both devices, yet for some reason it still shows up in “Out of Sync Items”
  • One device has a list of “Failed Items” it couldn’t obtain with the message “no connected device has the required version of this file” - but these exact files DO exist on the connected device and are not ignored…
7

Can you query the /rest/debug/file?folder=...&file=... endpoint for one of those files and post the result here please.

8

“Debugging disabled” :confused:

9

Can be enabled in web UI: Actions > Advanced > GUI > Debugging

10 
{
  "availability": [
    "THWBCSV-2AFJNJA-2OFU6DO-NK6H3HE-6GYXWKE-RPJE3AC-HILJHIF-7NUOLQH"
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2021-02-20T12:25:32+01:00",
    "modifiedBy": "PCZWK5P",
    "mustRescan": false,
    "name": "3-resources/.gitignore",
    "noPermissions": false,
    "numBlocks": 1,
    "permissions": "0644",
    "sequence": 581133,
    "size": 40,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "PCZWK5P:1617112127"
    ]
  },
  "globalVersions": "{{Version:{[{PCZWK5P 1617112127}]}, Deleted:false, Devices:{THWBCSV}, Invalid:{}}, {Version:{[]}, Deleted:true, Devices:{7777777}, Invalid:{}}}",
  "local": {
    "deleted": true,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2021-04-14T13:00:20+02:00",
    "modifiedBy": "5HJMFOJ",
    "mustRescan": false,
    "name": "3-resources/.gitignore",
    "noPermissions": false,
    "numBlocks": 0,
    "permissions": "0644",
    "sequence": 1819328,
    "size": 0,
    "type": "FILE_INFO_TYPE_FILE",
    "version": []
  }
}

Interestingly, THWBCSV is my phone, which is currently not online.

Querying for that file on my laptop (PCZWK5P), where it does exist, only produces an empty response :thinking:

11

And a scan doesn’t change that? Then Syncthing somehow doesn’t “see” it, which is basically impossible (or it’s ignored, but you said not). You could enable fs and scanner debugging when scanning, maybe there’s a clue in there.

12

Output changed after connecting my phone:

{
  "availability": [
    "THWBCSV-2AFJNJA-2OFU6DO-NK6H3HE-6GYXWKE-RPJE3AC-HILJHIF-7NUOLQH",
    "7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4"
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2021-02-20T12:25:32+01:00",
    "modifiedBy": "PCZWK5P",
    "mustRescan": false,
    "name": "3-resources/.gitignore",
    "noPermissions": false,
    "numBlocks": 1,
    "permissions": "0644",
    "sequence": 581133,
    "size": 40,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "PCZWK5P:1617112127"
    ]
  },
  "globalVersions": "{{Version:{[{PCZWK5P 1617112127}]}, Deleted:false, Devices:{THWBCSV, 7777777}, Invalid:{}}}",
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2021-02-20T12:25:32+01:00",
    "modifiedBy": "PCZWK5P",
    "mustRescan": false,
    "name": "3-resources/.gitignore",
    "noPermissions": false,
    "numBlocks": 1,
    "permissions": "0644",
    "sequence": 2118019,
    "size": 40,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "PCZWK5P:1617112127"
    ]
  }
}

Now there’s a device 7777777 listed which doesn’t even exist?

13

Got a little error after querying on my laptop (where the file exists but somehow isn’t indexed):

Fatal error: data DebugGlobalVersions(3-resource/.gitignore): key not found
14

That’s an already fixed bug on the debug endpoint where a “not found” error is incorrectly regarded as a fatal error.

77777 is the local device ID, regardless of what the device id of the local device is.

Actually there needs to be something wrong with the querying on laptop/PCZWK5P: The output shows that the file has been modified by it, thus it must have a record of the file. Otherwise the others couldn’t show that info. Double check that the folder isn’t paused, you are querying the right Syncthing instance, right folder id, … please.

15

yes, yes, yes. The folder id is data on both and it is shared.

But I just realized something odd: The “Global State” equals the “Local State” on each device, even though it differs by over 10GB!
Somehow they aren’t sharing indexes, I guess?

16

I have no clue what’s going on here - both too much and too little information to grasp the situation. You can grab the stindex tool from here: https://build.syncthing.net/buildConfiguration/Syncthing_Tools/100071?buildTab=artifacts (guest login). The run stindex -mode idxck and see if it reports any problems.

17

stat ~/.config/syncthing/index-v0.14.0.db: no such file or directory (is another instance of Syncthing running?)

stopped syncthing (via systemd), still getting this error

18

Then your db/config directory is not at that (the default) location. Provide the path to the actual location as the last argument to stindex

19

Database nowadays has three default locations:

https://docs.syncthing.net/users/config.html

If the database wasn’t manually set to a different location, check the other two default locations.

20

Laptop: 5 block list entries out of 14297 needs GC

PC: 1 block list entries out of 12945 needs GC

It was in XDG_DATA_HOME as it should be, weird that stindex doesn’t use that as default.