I cut these from the logs. I will have a go with the grep command in a moment. If we can work it out without the whole log I would prefer, but if we can’t then I will look to share them
My version of grep doesn’t have -P. Using -E gives lines with the file as well as any line containing INFO, and there is a lot of noise in the files with filenames containing INFO. This is only 27k though…
What’s missing in dirac is any mention of an index for that file from geo-fun. To check that this is indeed the case (though it’s likely given the repeoted need folder ... lines run grep -E 'insert;.+DZGK2SA.+20-29_Interests/21_Art/21.03_Flowers/Daffodil.ai.
On geo-fun, please grep for missing file for sequence number.
If that doesn’t yield anything we are in a bit of pickle because there is no file level logging when sending indexes, so you’d have to run a custom binary for further debugging. Would you be ok with that? It can be built on official CI from an official github branch to increase the thrustworthiness
It looks like on dirac the file is present in the db but not in the sequence “bucket”, which is used for sending indexes. Can you build
Upgraded your trust level.
Doable, would per PM here on the forum be ok? Also can you build go binaries? We have a tool to check database integrity, but unfortunately it’s not part of release bundles. If yes, could you build https://github.com/syncthing/syncthing/tree/master/cmd/stindex and run it like stindex -mode idxck *pathToYourConfig* please.
Ok, confirmed, the issue is a missing sequence entry in the db, i.e. the error occurred earlier. Meaning unfortunately new logs can’t tell us how it happened.
Do you have past logs? And if so, can you check if there was ever a panic.
Other than that I am out of ideas…
If you don’t want to touch the files, to get rid of this you’d have to remove and readd that folder (i.e. hash everything anew).
The sequence entries in the db are missing on geo-fun for reasons unknown. Therefore they are not sent in indexes which then causes the remote out-of-sync display (being out of sync on both devices is a red herring, that’s because it’s a conflict that would be resolved if the index was sent).
How does it know it’s a conflict if it lost it’s own entries? Or are you saying they exist locally, but weren’t sent or sent but lost on the receiving side?
In Troubleshoot a few items not syncing the db was checked, and it was confirmed that on device geo-fun for the files out of sync the sequence entry in the db is missing (the “normal” fileinfo entry is there). Therefore it doesn’t send these files to dirac. Dirac has the loosing conflict copy. So now dirac thinks geo-fun is out of date because it doesn’t have the file at all (no index sent) and geo-fun correctly reports dirac as out of sync because it has the loosing conflict. Now if the index were sent, geo-fun would pull the file and resolve the conflict.
How it ended up in this situation is a complete mystery to me (except for an unlikely panic or whatever that left the db in an inconsistent state, but that’s pretty unlikely for ~700 files I’d say).
Jan 10 15:56:14 hal syncthing[20958]: panic: runtime error: slice bounds out of range
Jan 10 16:03:12 hal syncthing[3477]: panic: runtime error: slice bounds out of range
Jan 10 16:31:41 hal syncthing[3926]: panic: runtime error: slice bounds out of range
Jan 10 16:44:52 hal syncthing[5532]: panic: runtime error: slice bounds out of range
Jan 10 16:52:12 hal syncthing[6289]: panic: runtime error: slice bounds out of range
Jan 10 16:57:18 hal syncthing[6743]: panic: runtime error: slice bounds out of range
Jan 10 17:00:44 hal syncthing[7069]: panic: runtime error: slice bounds out of range
Jan 10 17:04:05 hal syncthing[7319]: panic: runtime error: slice bounds out of range
Jan 10 17:06:36 hal syncthing[7552]: panic: runtime error: slice bounds out of range
Jan 10 17:09:46 hal syncthing[7740]: panic: runtime error: slice bounds out of range
Jan 10 17:12:06 hal syncthing[7961]: panic: runtime error: slice bounds out of range
Jan 10 17:14:48 hal syncthing[8143]: panic: runtime error: slice bounds out of range
Jan 10 17:17:02 hal syncthing[8336]: panic: runtime error: slice bounds out of range
Jan 10 17:19:53 hal syncthing[8510]: panic: runtime error: slice bounds out of range
Jan 10 17:22:41 hal syncthing[8715]: panic: runtime error: slice bounds out of range
Jan 10 17:24:59 hal syncthing[8913]: panic: runtime error: slice bounds out of range
Jan 10 17:27:39 hal syncthing[9088]: panic: runtime error: slice bounds out of range
Jan 10 17:30:02 hal syncthing[9281]: panic: runtime error: slice bounds out of range
Jan 10 17:32:18 hal syncthing[9472]: panic: runtime error: slice bounds out of range
Jan 10 17:34:34 hal syncthing[9650]: panic: runtime error: slice bounds out of range
Jan 10 17:36:52 hal syncthing[9820]: panic: runtime error: slice bounds out of range
Jan 10 17:39:09 hal syncthing[9993]: panic: runtime error: slice bounds out of range
Jan 10 17:41:23 hal syncthing[10169]: panic: runtime error: slice bounds out of range
Jan 10 17:44:01 hal syncthing[10339]: panic: runtime error: slice bounds out of range
Jan 10 17:46:16 hal syncthing[10528]: panic: runtime error: slice bounds out of range
Jan 10 17:48:37 hal syncthing[10698]: panic: runtime error: slice bounds out of range
Jan 10 17:51:05 hal syncthing[10875]: panic: runtime error: slice bounds out of range
Jan 10 17:53:37 hal syncthing[11060]: panic: runtime error: slice bounds out of range
Jan 10 17:55:54 hal syncthing[11248]: panic: runtime error: slice bounds out of range
Jan 10 17:58:37 hal syncthing[11420]: panic: runtime error: slice bounds out of range
Jan 10 18:00:57 hal syncthing[11615]: panic: runtime error: slice bounds out of range
Jan 10 18:03:18 hal syncthing[11804]: panic: runtime error: slice bounds out of range
Jan 10 18:05:33 hal syncthing[12181]: panic: runtime error: slice bounds out of range
Jan 10 18:07:52 hal syncthing[12389]: panic: runtime error: slice bounds out of range
I deleted the folder at both ends and created a new folder with a different id, but pointing at the same locations within the file system. This has resolved the issue.
As background, on geo-fun I originally installed the version that is in the Ubuntu 18.04 repositories. I also installed the snap version on dirac. The snap version didn’t have a version id, and geo-fun was complaining about not being able to sync symlinks. So I shut down those versions and installed all from the apt repository.
I thought that I had deleted all of the config and database directories, but maybe I had missed something. I am not sure if this was a contributing factor, but it provides some context in case anyone else is having problems.