(Long-time ST user. Thanks for a great piece of software!)
I’m very excited to see the untrusted device feature and was testing it out for my use-case.
Imagine for a specific shared folder you have a number of trusted clients and an untrusted server as a central always-on and backed-up node.
Worst case, assume all clients and the server suffer data loss. Is there a way to recover the untrusted server first, directly from a backup of the encrypted folder, and then have the trusted clients pull from that node?
When I tried this locally I would always end up with “unexpected items” on the untrusted machine and a “global state” of 0 files.
I tried ensuring that the token file was preserved and also tried setting up the new share with the same Folder-ID.
I am aware that there is a decryption tool that would regenerate the unencrypted folder from the encrypted one but this would require the untrusted server to temporarily have a decrypted version of all files.
Yes, I think it’s impossible, and I don’t think it’s planned, nor do I think it’s worth the complexity, as it would be a fairly large change.
It’s quite nice that syncthing can easily rebuild the database, it means that it’s own data is not that important. It still “holds” for encrypted folders, it’s just that the data has to be decrypted in order for it to rebuild the state.
The encrypted nodes simply don’t have the encryption keys to generate a database, that would pass encryption signature checks, unless syncthings database data became part of the file data on disk, but at that point, you might as well just back up syncthings database, so you don’t need to rebuild the database, given you’re already backing up the encrypted data.
TL;DR, in a scenario like this you would transfer the encrypted files to a trusted node, decrypt them, and boot strap stuff that way.
A quick test I forgot to make on my side was to wipe the untrusted db with configs still in place and let ST rebuild. Indeed, as you predicted, all existing encrypted files were considered unexpected and all files were re-downloaded from trusted clients.
Either way, the untrusted feature finally supports my particular use-case, so thank you a lot!
It is in fact technically possible to recover at least some of the database from the encrypted data set; each encrypted file has a copy of the required metadata attached to the end. However there’s currently no tool to do so. (There’s nothing like this for directories though, so it wouldn’t be complete.)
Folders aren’t not disk other than the metadata in the database. The offline decrypt will thus not restore the original directory permissions, nor empty directories, unfortunately. (If I remember correctly. I became slightly unsure while typing this, but on mobile so can’t be bothered to verify.)
Just came back to a computer:
Folders (and symlinks) are on disk, but as folders. Meaning there’s no metadata trailer (no file to add a trailer to). The decrypt tool could thus restore empty directories, but indeed not original permissions. It doesn’t do that though, it skips all non-file items.
We could store directories and symlinks as empty files, thus being able to add the encrypted metadata to them. Needs some thinking about what the ramifications of such a change would be - I am too tired for that now.
Just for my own curiosity I tried it out and, as you said, empty folders do indeed hit the disk of the untrusted client but are not recovered by the decrypt command.
I can’t speak for other people’s need to store and recover empty folders but it does seem to be a more complete solution.
I would personally be perfectly happy with a “file only” recovery but I do also value knowing that between trusted clients, the shares are perfectly identical.
At the end of the day, it’s up to someone (you guys) to balance what makes most sense for the larger user-base.
Keep in mind none of the above discussion relates to what happens during normal syncthing syncing. Between trusted clients, the results are indeed identical, empty directories and permissions and all.
The discussion here is only about what happens when all the trusted devices are gone, the database on the untrusted device is also gone for some reason (so you can’t decrypt by adding a new trusted device), and all you have to work with are the encrypted file contents. We can restore from those, but then not all nuances are preserved. I think this is in an acceptable state already, trade-off wise.
I only meant that if users are accustomed to the on-disk content being a complete description of their data, as is naturally the case on trusted clients, they might incorrectly assume the same is true for untrusted clients.
My past experience with ST and file-based encryption layers was the reason I initially assumed that was the case as well.
Either way, the existing solution is perfectly adequate for my uses so thank you again!
