Issues with Out Of Sync items on macOS and untrusted device

Similar issue that looks like it has the same root cause: How can I resolve "Out of Sync" error that's not going away by itself?

I setup a ignore rule for .DS_Store files on mac to exclude those, but now both of my machines are having issues fully syncing to an untrusted device

On my untrusted device I constantly get a out of sync item that I can’t resolve, and a ton of files that are locally changed

I tried the ‘revert local changes’ button but it doesn’t do anything. Also paused all the syncing before clicking it.

On the other side (my 2 macs), it looks like this:

The files it reports are almost always folders or .DS_Store files

Looking at that other thread, I think what happened is that ignoring .DS_Store files caused a bunch of empty folders to get synced without any files in it. I have since wiped those DS_Store files with a quick find . -name '.DS_Store' -type f -delete but the problem persists. I have also added a new ignore rule (?d).DS_Store to allow syncthing to delete folders when only a DS_Store remains.

I have a feeling that having the untrusted file be encrypted is what’s causing some of this, because the device can’t see the filename / filecontent (though no idea how syncthing actually works). I am also confused why Revert Local Changes seemingly doesn’t do anything

Any ideas how to fix this? Is there I command I can use to force overriding the remote store with whatever my device currently has, ala completely forcing a one-way re-upload?

Sync between my macs is fine and at 100%. Just to my untrusted device things got messy.

Syncthing v1.22.0

It’s difficult to say what exactly is going on, to be honest, especially since there appear to be a lot of files that are out-of-sync. I’d suggest to try using the “Revert Local Additions” button first, and then check the logs for what exactly happens after doing so to find out why the folders can’t be removed automatically.

I tried that already and no error shows up in the logs, but nothing happens either

For example:

[EZXVL] 2022/10/21 19:01:12 INFO: Reverting unexpected items in folder “folder name” (folder-id) (receive-encrypted)

And that’s about it. Can I force it to wipe away local changes?

/EDIT: I see there is a reset POST command: POST /rest/system/reset — Syncthing documentation

Would wiping the database fix this issue? I’d assume not so much because the local files would still be present…

What devices exactly are involved in the synchronisation here? Is it all Macs or rather a mix of different hardware and operating systems?

2 macs, 1 linux host running the container inside kubernetes (untrusted). Also not all folders having this issue, just the 2 earlier ones when I had the ignore rule.

I think at this point I’m ok with wiping the remote store and trying from scratch again, is there any good way to do it, or to force syncthing to throw away local changes? Like that button, but more force-ey?

You don’t necessarily need to re-do the whole configuration from scratch. I’d try using https://docs.syncthing.net/users/syncthing.html#cmdoption-reset-deltas first, and if that doesn’t work, then https://docs.syncthing.net/users/syncthing.html#cmdoption-reset-database. Just in case, I’d also in advance set all folders to “Receive Only” on the device where you’re going to perform these operations.

1 Like

Just to confirm, do you recommend doing these commands on the server that has the ghost files / weird state, or on my client that is syncing to the server?

-reset-deltas is a non-desctructive operation, so I’d suggest to start with this one. You can try it on the server first, and if the issue remains, on other devices as well. When it comes to -reset-database, you should proceed with caution, making sure that everything is set to Receive Only (and I’d also pause all connections to other devices too) before performing it. The same procedure applies here though — start with the server, then move to other devices if the issue hasn’t been resolved.

Okay I tried to do some debugging. Today I noticed that a new folder I created just yesterday had exactly the same issues, even though it had just 4 files and 1 folder in it!!

Out of all the shares I created yesterday, only the share that is unencrypted and without ignore rules did not run into the locally changed files issue

I created a bunch of shares with and without ignore rules, moved stuff around and tested if I could replicate the error and yup I was. It always happens when I shut down one of the pods (in kubernetes world, a thing running a container) mid-sync, to simulate what kubernetes would do if it were to move it to a different node

I think because the share is encrypted, syncthing might have some issues resolving what happened and picking up state. Unencrypted shares pick up just fine when the container suddenly shuts down

So to summarize:

Happens when the container shuts down mid sync on a untrusted device. Ignore rules and DS_Store aside, this happened on all test folders, besides the unencrypted one

Now going to focus on resolving the local issues

All right, I had more time on hand (rip my sunday…) here are some more comments and thoughts

  • -reset-deltas didn’t do anything, the folder stayed out of sync
  • -reset-database looks promising at first, but after it re-synced to 100% the problem came back, probably due to the files on disk being still identical to before:

Server:

Client:

I did a bunch of pausing, Revert Local Changes, resuming, adding versioning and eventually got rid all local errors, but once I restarted the container, a bunch of new ones appear.

One theory why this is happening (don’t have any devices besides my server to test), is that I am mounting a network share for syncthing that has eventual consistency, so from what I understand it is possible that between restarts the state might be slightly different before everything catches up. But the errors don’t go away eventually so not sure if this theory holds up. I also can’t test if the issue persists without the drive currently

Right now, no matter how healthy the state is, once I restart the container, all encrypted folders will have locally changed items (even though they are set to receive only encrypted)

I have tested disabling p2p sync between my devices and sync only through my server node, to see if I could notice any data inconsistencies and it looks fine, despite server reporting locally changes files. I don’t feel confident about this yet though and will do more tests to see if the content of synced folders through the server is actually identical…

I am unsure what to do here to be honest. Some options

  • Ignore the server state errors and just trust that it’ll work, because server is set to receive only anyway. Syncthing on the clients works fine and reports 100% synced, and I haven’t seen any data inconsistencies between the clients yet. Clients also p2p sync between them
  • Don’t use untrusted device, and instead see if I can move the encryption to the mounting layer instead
  • Give up on using network mounted drives, but that’s my entire point of using Syncthing on the server

If anyone has thoughts, please share :slight_smile:

Just one last update on this before I call quits for today

I initialized an empty git repo into the folder and committed everything, just to see that stuff actually is identical on the other side when only syncing through my server, with p2p disabled. So far it is, even after purposely killing the container mid-sync, and having a bunch of “Locally Changed Items”

So far it looks like it does actually end up syncing correctly, even with the untrusted device server having unexpected items

Going to do more tests and track changes with git

I’ve given up. I was constantly running into issues, even after switching mounting solutions and trying different configurations

Instead, I removed all the untrusted device features and mounted the drive with rclone crypt to be e2ee, so I moved the encryption layer from syncthing to underlying mounting solution. Performance isn’t that great but hopefully that’s going to be it with syncthing locally changed items problems

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.