"Local Additions" when one-way syncing TWRP backup from Android to Windows

I have a problem with syncing a TWRP backup folder from Android to Windows. The folder is set to “Send Only” in Android, and then shared to Windows. In Windows, I accept the folder setting it to “Receive Only”.

However, after the syncing finishes, the Windows side reports “Local Additions”. Pressing the “Revert Local Changes” button ends with

2020-04-09 11:35:25 Revert: directory is not empty; files within are probably ignored on connected devices only

even though there are no ignores set on neither of the two sides.

This is how the situation looks in Android,

and in Windows,

with the “Locally Chnages Items” reporting

Now, the actual TWRP folder structure in Android is

TWRP/.stfolder
TWRP/.twrps
TWRP/BACKUPS
TWRP/BACKUPS/41038b4f630460b9
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/boot.emmc.win
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/boot.emmc.win.sha2
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/data.ext4.win
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/data.ext4.win.sha2
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/data.info
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/recovery.log
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/system.ext4.win
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/system.ext4.win.sha2
TWRP/BACKUPS/41038b4f630460b9/2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k/system.info

and in Windows is

TWRP\.stfolder
TWRP\BACKUPS
TWRP\BACKUPS\41038b4f630460b9
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\boot.emmc.win
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\boot.emmc.win.sha2
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\data.ext4.win
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\data.ext4.win.sha2
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\data.info
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\recovery.log
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\system.ext4.win
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\system.ext4.win.sha2
TWRP\BACKUPS\41038b4f630460b9\2020-03-30--02-00-39_slim_n5120-userdebug_444_KTU84Q_8255_test-k\system.info

which shows that the missing file is actually

TWRP/.twrps

I have tried to copy the file manually, but not only does it not change anything, but also clicking the “Revert Local Changes” button simply deletes it. The file itself is just a configuration file for TWRP itself.

What can I do to try to debug the problem and find the cause of it? I have been syncing other folders between these devices with no such issues, so something seems to be wrong with this specific case, but I myself have no idea what it may be.

The reported locally changed items are clearly incorrect, I’d start investigating there. Can you please post the output of https://docs.syncthing.net/rest/db-file-get.html for (one of) those files, ideally on both devices but one might already give pointers.

1 Like

Windows:

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": true,
    "localFlags": 8,
    "modified": "2020-04-04T16:07:09.102862+09:00",
    "modifiedBy": "D4DZUGP",
    "mustRescan": false,
    "name": "BACKUPS",
    "noPermissions": true,
    "numBlocks": 0,
    "sequence": 1,
    "size": 128,
    "type": "DIRECTORY",
    "version": [
      "D4DZUGP:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": true,
    "localFlags": 8,
    "modified": "2020-04-04T16:07:09.102862+09:00",
    "modifiedBy": "D4DZUGP",
    "mustRescan": false,
    "name": "BACKUPS",
    "noPermissions": true,
    "numBlocks": 0,
    "sequence": 1,
    "size": 128,
    "type": "DIRECTORY",
    "version": [
      "D4DZUGP:1"
    ]
  }
}

Android:

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2020-03-30T07:00:39.96000002Z",
    "modifiedBy": "MN2XAEL",
    "mustRescan": false,
    "name": "BACKUPS",
    "noPermissions": true,
    "numBlocks": 0,
    "sequence": 1,
    "size": 128,
    "type": "DIRECTORY",
    "version": [
      "MN2XAEL:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2020-03-30T07:00:39.96000002Z",
    "modifiedBy": "MN2XAEL",
    "mustRescan": false,
    "name": "BACKUPS",
    "noPermissions": true,
    "numBlocks": 0,
    "sequence": 1,
    "size": 128,
    "type": "DIRECTORY",
    "version": [
      "MN2XAEL:1"
    ]
  }
}

I forgot about the most basic info: What Syncthing version are you on?

And I am sorry for not providing that basic information :sweat:.

It is Syncthing 1.4.2 in Windows and Syncthing 1.4.0 in Android.

Then please upgrade android (should be available in the play store, f-droid usually follows a few days after). The information from /rest/db/file you provided above (thanks) shows that the index entry from android is missing on windows. There were issues fixed in 1.4.1 that can cause this. So before investigating further I’d like to be sure that doesn’t resolve things for you.

1 Like

Thank you. All my Android devices run either Syncthing or Syncthing Fork from FDroid, and the current version there for both is still 1.4.0. I will wait for the update to be released, install it, and then report here again.

1 Like

I have updated Syncthing to 1.4.2 and the folder seems to sync fine right now :slightly_smiling_face:.

I did have to delete and re-add it, because otherwise one of the files was still marked as “Locally Changed”, but after that everything has been pulled from the phone again, and is marked “Up to Date” on both devices.

I did also get this error on Android after upgrading from 1.4.0 to 1.4.2, but other than that, there do not seem to be any issues with the synchronisation.

1 Like

Did you get that error repeatedly or once only? And I assume you don’t have the database from before “delete and re-add” anymore, right?

If he’s using the fork and has made an export settings before from the app he might still have it on the internal sdcard.

The device is using the Fork, and is also rooted, so getting the files is not an issue, but unfortunately I did not think about doing a backup before the “delete and re-add”.

I do have a backup from last night when everything was still on Syncthing 1.4.0, i.e. as described in the very first post of this thread, especially since the folder itself has not changed since then. Would that be of any use?

As for that error, I got it only once, right after upgrading from 1.4.0 to 1.4.2.

2 Likes

Then it’s ok, no need for the db - thanks.

I am writing again, because I have discovered another problem with a different folder. I could create a new topic, but the problem this time is also with a TWRP backup, just on a different Android device, so I thought that maybe the two are somewhat related.

Anyhow, the problem is that global and local states do not match on Android.

GET /rest/db/status

{
  "errors": 0,
  "globalBytes": 653302770,
  "globalDeleted": 0,
  "globalDirectories": 3,
  "globalFiles": 10,
  "globalSymlinks": 0,
  "globalTotalItems": 13,
  "ignorePatterns": false,
  "inSyncBytes": 653302770,
  "inSyncFiles": 10,
  "invalid": "",
  "localBytes": 653303730,
  "localDeleted": 0,
  "localDirectories": 3,
  "localFiles": 11,
  "localSymlinks": 0,
  "localTotalItems": 14,
  "needBytes": 0,
  "needDeletes": 0,
  "needDirectories": 0,
  "needFiles": 0,
  "needSymlinks": 0,
  "needTotalItems": 0,
  "pullErrors": 0,
  "sequence": 14,
  "state": "idle",
  "stateChanged": "2020-04-19T04:19:07.292509308Z",
  "version": 14
}

and on Windows

GET /rest/db/status

{
  "errors": 0,
  "globalBytes": 653302770,
  "globalDeleted": 0,
  "globalDirectories": 3,
  "globalFiles": 10,
  "globalSymlinks": 0,
  "globalTotalItems": 13,
  "ignorePatterns": false,
  "inSyncBytes": 653302770,
  "inSyncFiles": 10,
  "invalid": "",
  "localBytes": 653302770,
  "localDeleted": 0,
  "localDirectories": 3,
  "localFiles": 10,
  "localSymlinks": 0,
  "localTotalItems": 13,
  "needBytes": 0,
  "needDeletes": 0,
  "needDirectories": 0,
  "needFiles": 0,
  "needSymlinks": 0,
  "needTotalItems": 0,
  "pullErrors": 0,
  "receiveOnlyChangedBytes": 0,
  "receiveOnlyChangedDeletes": 0,
  "receiveOnlyChangedDirectories": 0,
  "receiveOnlyChangedFiles": 0,
  "receiveOnlyChangedSymlinks": 0,
  "receiveOnlyTotalItems": 0,
  "sequence": 13,
  "state": "idle",
  "stateChanged": "2020-04-19T13:56:11.4548009+09:00",
  "version": 13
}

In reality, there are 10 files in the folder, so the local state on Android is wrong. Also, both devices run Syncthing 1.4.2.

This time, before doing any “delete and re-add”, I would like to ask whether there is anything I could do to debug this beforehand :wink:.

I have reset the database under Windows, and now the original problem has come back, but with a different device, which is the one with incorrect local state mentioned above.

Windows:

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": true,
    "localFlags": 8,
    "modified": "2020-04-09T20:07:10.7126784+09:00",
    "modifiedBy": "D4DZUGP",
    "mustRescan": false,
    "name": "BACKUPS",
    "noPermissions": true,
    "numBlocks": 0,
    "sequence": 1,
    "size": 128,
    "type": "DIRECTORY",
    "version": [
      "D4DZUGP:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": true,
    "localFlags": 8,
    "modified": "2020-04-09T20:07:10.7126784+09:00",
    "modifiedBy": "D4DZUGP",
    "mustRescan": false,
    "name": "BACKUPS",
    "noPermissions": true,
    "numBlocks": 0,
    "sequence": 1,
    "size": 128,
    "type": "DIRECTORY",
    "version": [
      "D4DZUGP:1"
    ]
  }
}

Android:

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2020-03-29T22:00:12Z",
    "modifiedBy": "EEYCH4R",
    "mustRescan": false,
    "name": "BACKUPS",
    "noPermissions": true,
    "numBlocks": 0,
    "sequence": 11,
    "size": 128,
    "type": "DIRECTORY",
    "version": [
      "EEYCH4R:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2020-03-29T22:00:12Z",
    "modifiedBy": "EEYCH4R",
    "mustRescan": false,
    "name": "BACKUPS",
    "noPermissions": true,
    "numBlocks": 0,
    "sequence": 11,
    "size": 128,
    "type": "DIRECTORY",
    "version": [
      "EEYCH4R:1"
    ]
  }
}

Is there anything I can do about this apart from “delete and re-add”?

That was a bit too quick. For local/global state discrepancies -reset-deltas has a good chance of fixing the problem (and if it doesn’t, we have something to debug). Resetting the db is more likely to produce inconsistencies, more resource intensive and destroys all evidence.

As to the new issue, I’ll have a look this week.

I am sorry for that, but I just wanted to see whether resetting the database would make any difference with Unresponsive Web GUI - pause and resume is not registered, which is a much bigger deal for me compared to this TWRP thing.

I did make a backup of the database folder on Windows just before resetting it though, so please let me know if you need it. Also, I have not touched the database on Android.

The TWRP problem is neither that important nor urgent, so please take your time. Thank you for all the help anyway :slightly_smiling_face:.

I am refreshing the topic, as I wanted to say that the problem persists, and I just cannot find any clues, why the folder state ends up inconsistent like that :slightly_frowning_face:. Also, it seems to affect all folders, which are being shared between Android and Windows, not just the TWRP folder.

a) Folder TitaniumBackup is reported “Up to Date” on both sides.

Android:
{
  "globalBytes": 1266841437,
  "globalFiles": 428,
  "globalTotalItems": 482,
  "inSyncBytes": 1266841437,
  "inSyncFiles": 428,
  "localBytes": 1266841437,
  "localDeleted": 54,
  "localFiles": 428,
  "sequence": 536,
  "stateChanged": "2020-04-26T01:55:16.776838219Z",
  "version": 536
}
Windows:
{
  "globalBytes": 1271766400,
  "globalFiles": 430,
  "globalTotalItems": 484,
  "inSyncBytes": 1271766400,
  "inSyncFiles": 430,
  "localBytes": 1271766400,
  "localDeleted": 52,
  "localFiles": 430,
  "receiveOnlyChangedBytes": 0,
  "receiveOnlyChangedDeletes": 0,
  "receiveOnlyChangedDirectories": 0,
  "receiveOnlyChangedFiles": 0,
  "receiveOnlyChangedSymlinks": 0,
  "receiveOnlyTotalItems": 0,
  "sequence": 534,
  "stateChanged": "2020-04-26T10:55:01.1746572+09:00",
  "version": 534
}

Android-TitaniumBackup.txt (658 Bytes) Windows-TitaniumBackup.txt (862 Bytes)

Apart from the inconsistent state, the problem here is that in reality there are 443 files in the folder on Android rather than 428 or 430, as reported in Syncthing. Rescanning the folder does nothing, but there are files stuck in syncing on the Android device:

image

b) Folder TWRP is reported as “Local Additions” on Windows.

Android:
{
  "globalBytes": 1139074014,
  "globalDeleted": 0,
  "globalDirectories": 0,
  "globalTotalItems": 7,
  "inSyncBytes": 1139074014,
  "localBytes": 1139074014,
  "localDeleted": 0,
  "localDirectories": 0,
  "localTotalItems": 7,
  "sequence": 7,
  "stateChanged": "2020-04-26T01:48:27.869162187Z",
  "version": 7
}
Windows:
{
  "globalBytes": 1139074398,
  "globalDeleted": 1,
  "globalDirectories": 3,
  "globalTotalItems": 11,
  "inSyncBytes": 1139074398,
  "localBytes": 1139074398,
  "localDeleted": 1,
  "localDirectories": 3,
  "localTotalItems": 11,
  "receiveOnlyChangedBytes": 384,
  "receiveOnlyChangedDeletes": 0,
  "receiveOnlyChangedDirectories": 3,
  "receiveOnlyChangedFiles": 0,
  "receiveOnlyChangedSymlinks": 0,
  "receiveOnlyTotalItems": 3,
  "sequence": 26,
  "stateChanged": "2020-04-26T10:27:35.6769894+09:00",
  "version": 26
}

Android-TWRP.txt (642 Bytes) Windows-TWRP.txt (852 Bytes)

Reverting the “Local Additions” does not work:

2020-04-26 10:27:35 Revert: directory is not empty; files within are probably ignored on connected devices only

There are no ignores set, and the situation described above happens when sharing the folders from Android to Windows for the very first time, i.e. there were no existing files on Windows prior to the syncing operation.

Syncthing is 1.4.2 in both cases with https://github.com/syncthing/syncthing/commit/d3ed4de4ed6e2f061f35df4b270359820145f1c9 added on top of it, but as already described in this topic, the problem was happening way before that.

Are there any other logs that would help to dig deeper into the cause of this issue? I have really no idea what is going on here :dizzy_face:.

Edit:

Something very funky seems to be going on on Android.

I have added the same folder to Syncthing twice, without even sharing it, and the result is:

and then after changing the folder from “Send Only” to “Send and Receive”

In reality, there are 8 files and 3 folders inside the TWRP folder (excluding the .stfolder).

ls -la -R /sdcard/TWRP

/sdcard/TWRP:
drwxrwx--- root     sdcard_r          2020-04-26 00:54 .stfolder
-rwxrwx--- root     sdcard_r     1365 2020-03-30 07:19 .twrps
drwxrwx--- root     sdcard_r          2020-03-30 07:10 BACKUPS

/sdcard/TWRP/.stfolder:

/sdcard/TWRP/BACKUPS:
drwxrwx--- root     sdcard_r          2020-03-30 07:10 0009237466004f

/sdcard/TWRP/BACKUPS/0009237466004f:
drwxrwx--- root     sdcard_r          2020-03-30 07:19 2020-03-30--01-10-11 slim_i9100-userdebug 4.4.4 KTU84Q eng.warri

/sdcard/TWRP/BACKUPS/0009237466004f/2020-03-30--01-10-11 slim_i9100-userdebug 4.4.4 KTU84Q eng.warri:
-rwxrwx--- root     sdcard_r  8388608 2020-03-30 07:19 boot.emmc.win
-rwxrwx--- root     sdcard_r       48 2020-03-30 07:19 boot.emmc.win.md5
-rwxrwx--- root     sdcard_r 988367056 2020-03-30 07:19 data.ext4.win
-rwxrwx--- root     sdcard_r       48 2020-03-30 07:19 data.ext4.win.md5
-rwxrwx--- root     sdcard_r  1566375 2020-03-30 07:19 recovery.log
-rwxrwx--- root     sdcard_r 140751829 2020-03-30 07:10 system.ext4.win
-rwxrwx--- root     sdcard_r       50 2020-03-30 07:10 system.ext4.win.md5

which indicates that for whatever reason Syncthing is unable to correctly detect the files in the folder.

I think that we can call this one solved, at least for now.

I have updated the operating system from Android 4 to 6 on one device, and from Android 4 to 7 on the other, and now everything seems to be syncing smoothly.

I did not want to do it, as the devices are old and slow, and as such run usually much better under the older OS, but I just got fed up with all the syncing errors, and was also worried that I would eventually lose data because of them. I still need to update the last device, but I doubt that there will be any issues.

Anyway, thank you for all the help, and I will write back in case new problems arise (but I hope not :wink:).

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.