I have two Raspberry Pis that have for a long time separately and successfully received encrypted folders from either of two laptops.
As usual, when the latest RC became available I updated both RPis and both laptops. Soon after the laptops’ GUI were showing ‘Unexpected Items’ for the Received Encrypted folders and the laptop was reporting out-of-sync for those folders.
Reverting the unexpected items only resulted in them immediately reappearing and the out-of-sync folders remained. Curiously, there were no files shown in the out-of-sync modal only 3.6MB on both laptops.
I have since reverted all devices to V1.23.2 and after clearing the unexpected items all has returned to normal for the past 24 hours.
I’m wondering if others have notice this problem and if #8556 might be the culprit in V1.23.3-rc.1.
BTW, there was also a javascript error (yellow bar) appearing briefly after starting the GUI for each laptop. That has also gone after reverting to 23.2.
There’s been a change and a transition related to receive-encrypted folders, so this is 99% a problem with that.
I assume there’s nothing that looks out of the ordinary in the logs?
Could you make a backup of your config and db, and then collect information with the RC again? That would be very helpful.
Given the problem re-appears after doing a revert, I think the following would be the simplest to gain some first info:
Upgrade to the RC and wait until you are in that bad situation. Then take screenshots of the expanded folder and the (empty) list of unexpected items, just to get an overview. Then enable model and db error logging (going to be very noisy) and hit revert - wait until the you are back in the broken state. Then please share those sections of logs. If you don’t want to do it publicly but are comfortable with sharing it privately, you can send it by personal message here or to freisim93@gmail.com.
Actually that likely also goes beyond what I tested (simple setup of two devices, one encrypted one not): Could you describe the exact setup for the folder, i.e. what device shares with what other including encrypted or not.
RPi4 and RPi3B+ are running Bullseye 24/7 and are both updated regularly. They are independent of each other and don’t share ST folders except with both laptops. Laptops are running Xubuntu 22.10 and are regularly updated.
Folders are unencrypted and synced between both laptops. Shared folder between laptops and Pis are a mix of unencrypted (borg, kopia, restic and fsarchiver, all independently encrypted) and ST encrypted, such as documents and Thunderbird profile. The Pis independently use rclone to send their unencrypted folders to rsync.net and Mega.
The encrypted documents folder is synced by the Pis to my Android phone, where it’s unencrypted by ST.
That’s probably over-complicated (I’m trialing kopia and restic) but it works successfully.
I’ll try and get to your tests later today or tomorrow being careful not to screw up my setup.
Thanks. To double check my understanding I’ll reiterate some bits that might be relevant:
No encrypted folder on any of the Pis is shared directly with the other Pis, only with unencrypted devices. And the problems occur on all folders, not just ones involving android.
On the first point: correct, no shares between Pis.
My Android phone (V1.23.1) is not continuously connected, I can’t recall if it was connected after the other four devices were upgraded to rc.1 and when I noticed the persistent ‘Unexpected Items’.
However, pausing and unpausing folders on RPi4 to try to isolate the problem, I did notice that pausing the Document folder (shared Pis to phone) on the RPi4 caused the ‘out-of sync’ on the laptops to change to ‘up-to-date’, suggesting that that folder was the only problematic one. Unpausing it caused ‘out-of-sync’ again. I think that reverting Unexpected Items on the Thunderbird folder (not shared Pis to phone) worked. So Android might be involved.
I will ensure the phone is not connected and upgrade the other devices and carefully check what happens.
I thought I would try a brute force solution on the Receive Encrypted devices (RPi4 and RPi3): stop Syncthing, delete the database, upgrade to rc.1 and then restart Syncthing. At the same time I upgraded the two laptops to rc.1.
That seemed to work initially (all green up-to-date) until files changed and Unexpected Items appeared on the encrypted devices. Reverting those items and restarting Syncthing only continued the cycle. Even stopping and restarting Syncthing on the Pis caused Unexpected Items without altering any files.
Frustratingly, I now back to 1.23.2 on all four devices and peace returns. But what of the future?
Yeah I need something more here. I tried to reproduce and couldn’t. Either more detailed steps to reproduce the issue, or the logs I suggested or maybe just syncthing cli debug <folder-id> <path> for one of the locally changed files might give me some hint. Also in the screenshots local folders are always fine, only remotes show an issue, but you do describe issues on the local folder (which also is the more reliable/important status indication).
Oh and just in case: I am not saying there’s no issue, just that I fail to find it and thus need help to make any progress.
What will definitely go bad (just did that myself), is cycling between v1.23.2 and v1.23.3, due to the migration not being idempotent. However for me that fixed itself by reverting local changes and pausing/unpausing the folder (the latter I think is a known issue that somehow the revert doesn’t work properly and thus requires a folder restart (or maybe just scan ).
Also feedback from anyone else using receive-encrypted would be very welcome: Do you see similar problems or not?
Simple experiment: Upgrade RPi4 only to v1.23.3~rc.1. On laptop (v1.23.2) touch a file in a folder sync’ed receive encrypted to pi. Immediately, one unexpected item appears on the pi with Revert Local Changes button. Revert the change and the folder is up-to-date. This is easily reproducible.
I have captured the log during this process with modal and db enabled and sent it freisim93@gmail.com
I mean the results were the same, whether the unencrypted device (laptop) was on 1.23.2 or 1.23.3-rc.1 and the encrypted device (RPi4) was on 1.23.3-rc.1.
To be clear, the RPi4 immediately reported Unexpected Item, reverting cleared the Unexpected Item, but touching the file again on the laptop created the same result on the RPi4, ie Unexpected Item. Just one, because the ‘touch’ was only one file.
It is, thanks for that and the logs. Unfortunately they still have me puzzled as to what’s happening. Could you also give me the output of this command please: syncthing cli debug file Gs4w4-k4eUb M.syncthing-enc/EU/RQ5KK0ITIUV46Q5PUSBN906LDA4VSIKBJO1DHF7F3FNSG9G. You can send it by email, but there’s not much point really, nothing sensitive in encrypted metadata.
Surely, i can’t be alone on the ‘bleeding edge’ here? Others would find this anomaly/bug or there is something awry with my setup that exposes the change to rc.1
You can enable debugging in the UI: Actions > Advanced > GUI > debugging
There has to be some edge case involved, otherwise I could reproduce. Also I wouldn’t expect that many people using both RCs and encryption (beta), and noticing issue straight way. Thanks again for doing that and reporting!
Maybe an edge case, as you say, but encryption of data on my Pis has been working for at least two years through many Syncthing versions up until the changes introduced in the latest RC. I’m not skilled enough to understand what changed and why but bugfix #8556 changing file sizes seems likely.
Is it possible to try a binary without #8556 to narrow down the cause between 1.23.2 and 1.23.3-rc.1?
I’ve also tried to eliminate complexity by using just two devices and minimising file changes by just using ‘touch’ to provoke a rescan.
There hasn’t been much happening since 1.23.2. And regardless, I’d still need debug info to find out what’s going on. Could you enable debugging and run that command please.
).