[SOLVED:because i tried to outsmart Syncthing] why are these files Out of Sync ? (with api db query result)

Tom · September 15, 2019, 10:42am

This file is OOS on A7JAAVJ, which is a W10 host set to Send Only.

JHCJAQL is Debian, Send & Receive.

{
  "availability": null,
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2017-12-18T16:05:11.49124+01:00",
    "modifiedBy": "JHCJAQL",
    "mustRescan": false,
    "name": "24/season 02/24.S02E01.Day.2_.8_00.A.M..-.9_00.A.M..720p.WEB-DL.DD5.1.H.264.H265.mkv",
    "noPermissions": false,
    "numBlocks": 1218,
    "permissions": "0755",
    "sequence": 8594,
    "size": 638368315,
    "type": 0,
    "version": [
      "A7JAAVJ:2",
      "JHCJAQL:4"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2017-12-18T16:05:11.49124+01:00",
    "modifiedBy": "JHCJAQL",
    "mustRescan": false,
    "name": "24/season 02/24.S02E01.Day.2_.8_00.A.M..-.9_00.A.M..720p.WEB-DL.DD5.1.H.264.H265.mkv",
    "noPermissions": false,
    "numBlocks": 1218,
    "permissions": "0755",
    "sequence": 8594,
    "size": 638368315,
    "type": 0,
    "version": [
      "A7JAAVJ:2",
      "JHCJAQL:4"
    ]
  }
}

Why the Out of Sync?

Should i Override Changes on the sender ?

Many thanks!!

calmh · September 15, 2019, 11:24am

Looks in sync to me. Is the output from the device where it’s out of sync?

Tom · September 15, 2019, 1:05pm

no, it was from the other side, not from the out of sync side. Meanwhile, I’ve removed the sync, and re-added it. Waiting on the 1st exchange to complete.

calmh · September 15, 2019, 3:32pm

Then it’s expected. If you want to see what differs, check on the side that is differing.

Tom · September 22, 2019, 9:25am

following up on this… ST is still deciding that files are out of sync, then transfers them, creates a sync-conflict file for each because the files already exist since i pre-seeded, I then delete the conflict file, and then the sync is fine.

Both sides are set to ignore permissions.

So, given the below examples, why are these files considered out of sync ? Why is ST transferring instead of re-using or copying from original ?

One example, db info from JHCJAQL to which the file is transferring.

{
  "availability": [
    {
      "id": "A7JAAVJ-ZVXYAH5-XJDMR76-ESC7HH4-LGQU6TH-KF45JZK-LE5AVCW-ZWVDUAB",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2013-09-11T01:15:40+02:00",
    "modifiedBy": "",
    "mustRescan": false,
    "name": "Night Train To Lisbon (2013)/Night.Train.To.Lisbon.2013.1080p.BluRay.DTS.x264-PublicHD.mkv",
    "noPermissions": false,
    "numBlocks": 49549,
    "permissions": "0644",
    "sequence": 18358,
    "size": 6494361735,
    "type": 0,
    "version": [
      "A7JAAVJ:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2013-09-11T01:15:40+02:00",
    "modifiedBy": "JHCJAQL",
    "mustRescan": false,
    "name": "Night Train To Lisbon (2013)/Night.Train.To.Lisbon.2013.1080p.BluRay.DTS.x264-PublicHD.mkv",
    "noPermissions": true,
    "numBlocks": 1549,
    "permissions": "0755",
    "sequence": 11455,
    "size": 6494361735,
    "type": 0,
    "version": [
      "JHCJAQL:1"
    ]
  }
}

Another example, db info from JHCJAQL to which the file is transferring.

{
  "availability": [
    {
      "id": "A7JAAVJ-ZVXYAH5-XJDMR76-ESC7HH4-LGQU6TH-KF45JZK-LE5AVCW-ZWVDUAB",
      "fromTemporary": false
    }
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2013-08-22T02:06:44+02:00",
    "modifiedBy": "",
    "mustRescan": false,
    "name": "Now You See Me (2013)/Now You See Me (2013) 720p.RERIP.PROPER.BLURAY.scOrp.mkv",
    "noPermissions": false,
    "numBlocks": 76069,
    "permissions": "0644",
    "sequence": 18363,
    "size": 9970413019,
    "type": 0,
    "version": [
      "A7JAAVJ:1"
    ]
  },
  "local": {
    "deleted": false,
    "ignored": false,
    "invalid": false,
    "localFlags": 0,
    "modified": "2013-08-22T02:06:44+02:00",
    "modifiedBy": "JHCJAQL",
    "mustRescan": false,
    "name": "Now You See Me (2013)/Now You See Me (2013) 720p.RERIP.PROPER.BLURAY.scOrp.mkv",
    "noPermissions": true,
    "numBlocks": 1189,
    "permissions": "0755",
    "sequence": 11457,
    "size": 9970413019,
    "type": 0,
    "version": [
      "JHCJAQL:1"
    ]
  }
}

Lucky for me this is on a LAN, so i’m not too bothered about it, it just takes much time to transfer all of it…

EDIT : forgot to add this screenshot, to indicate that ST seems to be actually transferring those files :

AudriusButkevicius · September 22, 2019, 12:29pm

You have large blocks enabled on one side but not the other.

Tom · September 22, 2019, 2:44pm

do you mean this parameter ? https://docs.syncthing.net/advanced/folder-uselargeblocks.html

I can’t even find the option in the GUI…?

Now that i’m just over half-way through letting the files transfer over the LAN, would it be a good idea of changing the option (provided i can even find it?) ?

EDIT : New in version 1.2.0: As of Syncthing 1.2.0 large blocks always enabled and the configuration option has been removed.

Then…how could 1 side have it set to disabled ???

Comparing the outputs of /rest/system/config on both sides, and I see no mention of uselargeblocks anywhere?

AudriusButkevicius · September 22, 2019, 3:25pm

Are they both running latest version of syncthing?

(Presumably because its missing on both configs it should be).

AudriusButkevicius · September 22, 2019, 3:30pm

Ok, the reason its transferring is because on one side the files were scanned before large blocks were on by default (so the original file is scanned in small blocks on one device and large blocks other), the fact they are the same does not matter as the chosen block size does not match so we have no way of verifying they are the same.

You can either touch the files causing syncthing to rescan them, or remove and readd the database on the older device where the files are scanned with small block size.

Tom · September 22, 2019, 7:28pm

As you deduced : yes, both 1.2.2

Again, correct. One side (send-only) dates back from 2015. It has updated automatically over the years so the index was recreated a few times when you guys made db changes. But not recently, to my knowledge.

On the older one ? It’s W10, it there a touch tool that’s reliable ?

I have a 3rd device in the mix, which I took offline because I noticed this funkyness happening. Also a Windows host. What would be the correct way of finding out the blocksize on that one, before adding it back into the group ?

AudriusButkevicius · September 22, 2019, 10:13pm

You can query the API, the block count is written there.

You can just rename the file, scan and rename it back, which would be am equivalent of touch.

Tom · September 23, 2019, 9:44am

For posterity’s sake :

On the old Send-Only Win10 node, which we concluded has many files that were originally added to index without large blocks, in the share’s root I am going to :

Powershell:
ls * -recurse | % { $_.LastWriteTime = Get-Date }

Caveat :

this will cause the entire share to be re-indexed.
any connected nodes will also be re-indexed.
some files will be briefly out-of-sync but no data will be transferred, all data will be re-used locally to generate a fresh copy of the files, and the block size will be set automatically (which is the entire purpose of this).

But the bright side is that all nodes will now be rid of non-automatic block sizes. Making future syncs to new nodes much faster, especially if the new node is seeded.

@AudriusButkevicius Some migration tool/function (which would re-index only those files with the non-auto block size), might have been nice?
The way i see it, i should now do the above for all shares, just to be sure there are no lingering old block sizes ??

AudriusButkevicius · September 23, 2019, 9:58am

This is only needed because you yourself pre-populated the folders out of bound instead of letting syncthing do it’s thing, so migration tool for a self induced problem is a waste of time in my book.

I would suggest remove the folder on all devices, add it back, let them converge, get on with your life. That’s the migration tool.

Tom · September 23, 2019, 10:05am

Brutally honest, but spot on. Thanks for having pointed out the cause though!

And a zillion thanks for the great software to you & all devs.

system · October 23, 2019, 10:05am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.