One item stuck in 'Out of Sync Items'


(tk83) #1

Hi,

I’ve got 2 Ubuntu Linux computers synced up perfectly, except for one single item that is permanently stuck in the Remote Devices->‘Out of Sync Items’. The folders under ‘Folders’ all show ‘Up to Date’.

This item is a directory called ‘local-xenial’ (a local apt repo). I’ve tried moving this directory to local-xenial.bak and then back again to trigger a resync, but not luck. I’ve also tried moving the directory away, creating it again with mkdir and moving all its files back in. This has also not worked.

It looks like the items is stuck in the Syncthing database somehow, since the Mod. Time is wrong on it anyway (see screenshot).


(Audrius Butkevicius) #2

This is a remote out of sync item, the cause of why it’s out of sync is probably visible on the device in question.


(tk83) #3

I’ve checked /var/log/syslog on the remote side (hostname timserver). I don’t see anything relevant (note that the chmod errors are something I’ve already fixed by changing ownership of these files to the same user that syncthing runs as):

Aug  7 08:40:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf"): dir chmod: chmod /home/srv/packages/localrepos/local-xenial/conf: operation not permitted
Aug  7 08:40:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/options"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/options: operation not permitted
Aug  7 08:40:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/distributions"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/distributions: operation not permitted
Aug  7 09:12:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf"): dir chmod: chmod /home/srv/packages/localrepos/local-xenial/conf: operation not permitted
Aug  7 09:12:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/distributions"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/distributions: operation not permitted
Aug  7 09:12:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/options"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/options: operation not permitted
Aug  7 10:16:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf"): dir chmod: chmod /home/srv/packages/localrepos/local-xenial/conf: operation not permitted
Aug  7 10:16:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/distributions"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/distributions: operation not permitted
Aug  7 10:16:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/options"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/options: operation not permitted
Aug  7 11:20:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf"): dir chmod: chmod /home/srv/packages/localrepos/local-xenial/conf: operation not permitted
Aug  7 11:20:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/options"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/options: operation not permitted
Aug  7 11:20:54 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/distributions"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/distributions: operation not permitted
Aug  7 11:38:52 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf"): dir chmod: chmod /home/srv/packages/localrepos/local-xenial/conf: operation not permitted
Aug  7 11:38:52 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/distributions"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/distributions: operation not permitted
Aug  7 11:38:52 timserver syncthing[10782]: [HXRFM] INFO: Puller (folder "localrepos" (uxlg9-benuv), file "local-xenial/conf/options"): shortcut chmod: chmod /home/srv/packages/localrepos/local-xenial/conf/options: operation not permitted

(Audrius Butkevicius) #4

Well they are still happening, so it seems your fix is not sufficient. Potentially the parent directory needs permissions fixed.


(tk83) #5

No sorry, I meant that those log messages are all that appear when I grep /var/log/syslog but that they’re all old messages from before I fixed the permissions problem, so aren’t relevant. The permissions are fine on both the directory and its parent.


(Simon) #6

Meaning on the timserver device the localrepos folder is showing as up-to-date, right? That means sync is all right but this device has incorrect information about the state of the other device. Resetting the index will probably help. If you want to help us find the cause, query the /rest/db/file endpoint on the file in question on both devices. See https://docs.syncthing.net/rest/db-file-get.html and this post and following ones for infos Other hosts all at "Syncing (95%, 0 B) No guarantees that it will help though, all previous samples were completely mysterious. If there is anything particular about that file and how you share it, please let us know - maybe something rings a bell.


(Audrius Butkevicius) #7

Can you actually post screenshots from both sides before you do that?


(tk83) #8

Thanks for both suggestions. Here’s the screenshot of the Syncthing interface on ‘timserver’, the previous screenshot was Syncthing’s web interface on ‘timvps’:


(Simon) #9

The folder panel isn’t visible, so this question still remains:


(tk83) #10

The ‘localrepos’ folder shows as ‘up to date’ in the Syncthing web interface on both machines


(Audrius Butkevicius) #11

Can you expand the folder with issues on both sides and then post thw screenshots from both sides for the folder in question?


(tk83) #12

timvps:

timserver:

Just let me know if there’s more info I should add


(tk83) #13

I also tried the REST API like @imsodin suggested, but wasn’t able to get it working:

curl -X POST -H "X-API-Key: abc123" http://localhost:8386/rest/db/browse?folder=default
404 page not found

(Audrius Butkevicius) #14

If you don’t have anything sensitive in the filenames, couls you zip up both databases from both devices and post them somewhere? It’s clearly a reporting bug, but we’ve been chasing it for ages.


(tk83) #15

The ‘localrepos’ folder is just a local apt repo so nothing sensitive, but the other Folder that’s synced between these 2 machines is my home directory. Is there any way to separate out just the databases for the ‘localrepos’ folder, without including the home directory folder?


(Audrius Butkevicius) #16

You could remove the folders and restart syncthing which would garbage collect the data about the folders you don’t want.


(tk83) #17

I’ll give this a try: I’ve edited config.xml on both machines and removed the Syncthing folder config for my home directory. All the database files (~/.config/syncthing/index-v0.14.0.db) with recent mod times still have the home directory’s files in them.

So at the moment, after restarting syncthing, the only Folder appearing in Syncthing is ‘localrepos’. How long would I need to let it run before it’ll create a new DB, or garbage collect?


(Audrius Butkevicius) #18

It should do it after a restart, so I would not expect the files to be there.


(tk83) #19

I left it running for over an hour, but it didn’t create a any new index files under ~/.config/syncthing/index-v0.14.0.db, just updated the same old ones that included the home folder’s stuff (I could easily tell by just opening them in less).

I stopped Syncthing again, renamed the existing index-v0.14.0.db/ on both machines and let it create new database files under that. But of course that new DB it’s created doesn’t have the problem - it shows ‘Up to Date’ under Remote Devices.


(Audrius Butkevicius) #20

Yeah, new database causes a retransfer of the index. Are you sure its not the db logs you find the filenames in? I am fine not to have the log.