Folders out of sync, even after rebuilding indexes

Support
1

I’m using Syncthing under Synctrazor on Windows (currently 0.14.9 and 1.1.11). I’ve got several devices syncing some large folders (one has 38000 photos, one has 2000 large files camcorder dumps).

In the last few months more of the folders have started showing “out of sync”. All are normal folders (no Masters). The Web GUI has the failed items count just showing the circle of dots going round and the out of sync items windows not actually changing. I’ve also had it go to the “can’t talk to syncthing” error too.

I decided to bite the bullet and shut everything down and recreate the indexes.

I started on my main file store PC and waited for everything to show in sync.

I then started up my editing PC and it’s showing some folders in-sync and others out of sync. It’s pulled down alot of missing files, but they still have the ~syncthing~xxxx.xxx.tmp filenames, but they have the full byte count so it looks like they have copied over. Others that are showing in the out of sync list actually exist and have the same byte count and modified date. Then there are some properly missing from the editing PC and refusing to come down (folders are created but empty)

Pasted image505×533 34.1 KB

2

The issue with the spinning “Failed Items”, but “Out of Sync”, is this:

Have a look at the “Out of Sync Items”. Are those files coming from a device which is not online?

3

Nope, they’re coming from a device that’s online (the file store pc). As I’ve rebuilt the indexes and only have these two nodes online there’s no knowledge of other files on other nodes (yet).

In fact some out of sync files already exist locally, so I’m not sure why they’re even in the list.

4

Anything in the log (that is, SyncTrayzor’s ‘console’ window)?

5

Nothing apart from some upnp errors periodically:

[SVKBT] 11:30:48 INFO: Invalid IGD response: invalid device UUID upnp-InternetGatewayDevice-1_0-2cb05d12f0ef (continuing anyway)

6

Well if there are out of sync items, there must be log messages about why in the logs.

7

The screenshot shows, that the folder is shared with multiple remote devices.

Files are created with the full size, so you cannot use the size as an indication of the transfer status.

8

Yep, but those devices are offline currently and haven’t been online since the indexes were rebuilt so these two nodes have no knowledge of any files they have, only that another node exists and is offline.

Here’s one of the large folders showing nearly everything out of sync but actually they aren’t.

What puzzles me is why there’s no progress on syncing. Even if the timestamps were wrong then shouldn’t it just process through them? There’s no significant disk or network activity i can see going on.

9

When it says its out of sync, it’s not trying to sync, and the errors explaining why should be in the console.

10

No errors in the console :slight_smile:

11

Ok, so if you had nodes A, B and C. C made changes that B and A were informed about yet they never synced them, they will show up as out of sync while C is down. Rebuilding A’s index has no meaning, as B will inform A about the changes as they connect, hence you need to rebuild both concurrently without a chance for them to connect to each other in between.

12

Hi,

As I said in my original post, I brought A online when B and C were both down and waited for it to finish indexing (showing everything in sync as expected)

I then brought B online for it to index while A is online. Some folders show in sync. Some folders were out of sync but pulled/pushed files until they were in sync.

You say Node B should have indexed offline, but I’m not sure I understand why. The behaviour I’ve observed on some folders is to merge what’s the global picture (from node A) with some additional files from Node B and upload/download them as necessary and eventually show up to date status.

On the out of sync folders, this process has begun (as seen by creation of some .tmp files) but not completed - it looks like it has stalled.

(C has never been online and has never informed A or B about any changes)

13

No, when you merge you merge A B and C, even tho C is offline, B knows what C had so the global picture potentially includes C’s files.

Anyway if things are failing, the logs will tell you why.

14

Do the following if its easier for you. Delete the folder from both sides simultaniously, restart simultaniously, and readd the folder.

15

Sorry, how does B know what C has when B has also built it’s index from nothing?

The logs don’t say why things are failing - is there a debug mode I need to enable?

16

Its explained in the docs. We’d be interested in model,scanner.

B will not know about C if its rebuilt, but as I said they have to be rebuilt concurrently to forget things, as otherwise one refreshes the others memory

17

They have been rebuilt concurrently. Well C hasn’t been rebuilt, but it’s still offline so is nothing to do with the sync issues.

I’m reindexing node B again with A (and C) offline with model,scanner debugging on. Is there a particular string I need to look for?

18

Well mainly strings related to the files which end up out of sync, from both sides.

19

Reindexing Node B doesn’t appear to be working. It’s created the index folder but the only files visible are the current, log, lock, manifest-000000 and 000001.log files (no ldb files).

I’ve got this in the syncthing log repeating over and over (for the last couple of hours).

[SVKBT] 2016/10/21 20:44:15.063956 walk.go:189: DEBUG: Walk \?\Z:\Movies-Temp [] current progress 1749004748/4121401086 at 23.6 MB/s (42%) [SVKBT] 2016/10/21 20:44:17.064070 walk.go:189: DEBUG: Walk \?\Z:\Movies-Temp [] current progress 1810676820/4121401086 at 24.2 MB/s (43%) . . [SVKBT] 2016/10/21 23:03:07.064519 walk.go:189: DEBUG: Walk \?\Z:\Movies-Temp [] current progress 1810676820/4121401086 at 0.0 MB/s (43%)

All the folders are showing unknown status and remote devices have no status showing.

These are the last few log entries before the repeating message above. I’ve no idea what these mean though so may be irrelevant.

[SVKBT] 2016/10/21 20:44:05.223393 rwfolder.go:302: INFO: Completed initial scan (rw) of folder GalaxyTab-Backups [SVKBT] 2016/10/21 20:44:05.223393 walk.go:318: DEBUG: rescan: File{Name:"", Type:FILE, Sequence:0, Permissions:00, ModTime:1970-01-01 00:00:00 +0000 GMT, Version:{[]}, Length:0, Deleted:false, Invalid:false, NoPermissions:false, Blocks:[]} 1459069628 -rw-rw-rw- [SVKBT] 2016/10/21 20:44:05.223393 walk.go:330: DEBUG: to hash: RAW-2016-03-27-Elise-Theo\DSC01485.ARW File{Name:“RAW-2016-03-27-Elise-Theo\DSC01485.ARW”, Type:FILE, Sequence:0, Permissions:0644, ModTime:2016-03-27 10:07:08 +0100 BST, Version:{[{SVKBTN2 1}]}, Length:16908288, Deleted:false, Invalid:false, NoPermissions:false, Blocks:[]} [SVKBT] 2016/10/21 20:44:05.224393 walk.go:318: DEBUG: rescan: File{Name:"", Type:FILE, Sequence:0, Permissions:00, ModTime:1970-01-01 00:00:00 +0000 GMT, Version:{[]}, Length:0, Deleted:false, Invalid:false, NoPermissions:false, Blocks:[]} 1452114303 -rw-rw-rw-

20

Nothing, this just means that it’s scanning. Given the rate is so low I suspect you are on a fairly weak device.