Encrypted nodes (please test)

[AudriusButkevicius]

low-powered and encryption does not go well you know.

[bademux]

yep encryption and hdd connected by usb2 can be a problem for someone, but as I said - for backup it should be ok to have ~2-3MB/s Or you suppose it will be mesured in KB/s ?

[bigbear2nd (Bob Hope)]

If the data ends up at the low powered device, it will not be decrypted…

So I guess it is as fast as the normal Syncthing syncspeed.

[Lennix]

@calmh

File name encryption seems mostly sane; possibly we need to add some hierarchy to it but that’s trivial. However, I can check if you have a given file or not by comparing the filename hashes. This is more difficult as I don’t know the directory prefix, but may still be an issue?

Best solution would be to generate a random IV and save it locally (in the DB). In case of desaster recovery the IV is saved within the filename. We currently have a problem with deep folder structures since they generate (too) long filenames. How does syncthing handle this without encryption? Where’s the limit?

@calmh

I’m adding some more high level notes here from the pull request, as there more eyes and discussion here;

---

• It’d be ideal if the encrypted side didn’t need to know it was encrypted. In fact, what happens if we don’t tell it and there’s a mismatch between the two sides in their assumption about the encryptedness?

Currently we need the eNode to stop scanning the local files. Once we send the encrypted hashes, the client doesn’t need to know that it’s receiving encrypted files.

@calmh

• There’s too much code duplication here between the rw and enc folder types. You don’t necessarily need to fix that, but before merging this I think we need to do some refactoring to break out common parts into something more sensible, so be prepared for that to happen. It’s been on the todo list for a while but this makes it more urgent.

The encFolder should become obsolete once the encrypted hashes are exchanged. On the other hand, we also need the cleartext hashes for the client to verify that it got the correct blocks. So there’s a little bit more logic necessary compared to the rwFolders.

@calmh

• Do symlinks survive the passage? Are they encrypted?

Haven’t tested that yet.

@calmh

• PBKDF per crypto operation is unnecessarily expensive; some other way here would be nice - use an IV and stash it somewhere?

Using an IV would mean going back to block cipher for the encryption. That would result in different block sizes on the client and the eNode, but is possible to realize.

@calmh

• This doesn’t encrypt the file content hashes, which means I can check if you have a given file by comparing the hashes with an unencrypted copy of that file.

That’s a good point, but should be no problem to encrypt the hashes as well.

@calmh

• This doesn’t hide file modification times and so on. Do we need this? I don’t see a reason we need to expose it, and the less data we give an attacker the better?

I don’t know how the OSes react if we set mod times like 14/58/9874. So encrypting the mod times needs to output sane timestamps which is the biggest hurdle I’d say. We could set all files to the same mod time and save the original onces encrypted somewhere else thou.

@calmh

• Do we want data verification on the encrypted devices? Spontaneously, I do, but I can’t say it’s super essential…

Does the underlying protocol check the transfers for error? Or is it relying on the block hashes? If that’s the case, we should check for data integrity on the eNode.

In summary:

• send the hashes of the encrypted files
• send the encrypted hashes of the files
• go back to block cipher and IV for file encryption and add support for unequal block sizes
• possibly encrypt the file information and save them somewhere on the eNode while saving the files with random modification times
• find some way to handle very long filenames or encrypt each part of the path separately
• use a random IV for the filename encryption and save it somewhere in the db in addition to the ciphertext
• possibly use a different key for the filename encryption to avoid a known-cleartext-attack (although it’s debatable that AES is vulnerable to those)

[Lennix]

Let me throw another idea into the round:

How about compressing the data before encrypting it. I saw there’s an option to compress data on packet level, but once we’re sending encrypted packets we’re rendering the compressing useless. Also it should work better on a larger dataset.

To make things a little easier we could compress on block level. But that would result in blocks ranging in different sizes. The index would need to save the size (and offset) of each block separately. Is that easily doable?

Also this should still be an optional feature, since it might eat alot into performance. There might be some testing necesarry, but IMHO especially regarding cloud backups, the bottleneck is still the upload bandwith of your connection and not so much CPU performance.

[calmh (Jakob Borg)]

I’m not sure there’s enough to gain by this to offset the extra complexity? The default metadata-only compression still makes sense for encrypted connections, and to be honest the compress-everything setting is seldom useful. Most large files they would benefit from compression are already compressed (with the notable exception of VM images I guess)… I’m fine with encryption being somewhat mutually exclusive with payload compression.

[Lennix]

@calmh

Most large files they would benefit from compression are already compressed (with the notable exception of VM images I guess)…

Since we’re transferring files on block level, every compressable file would benefit from compression. (At least files > Blocksize) This applies especially to documents, which are one of the primary backup targets (IMHO).

I’ve made some non-scientific tests and using the bundled lz4 compression can result in up to 35% reduced data. With a blocksize of 128kB thats about 40kB less to be transfered and saved.

The compression took on average 1ms for a full block, regardless of content.

The savings ranged from nothing to an average of 15kB up to 40kB.

Given 1ms for compression and an average of 15kB saved, results in roughly 100mbit/s saved. So if a user has a faster upload than 100mbit/s he would loose time using compression, although this only takes single-thread performance into account. (If I didn’t make any calculational errors)

Since we are changing the blocksize during encryption anyway, I think adding compression isn’t that more complicated and the results are quite impressive. Also compression has the added benefit, that it adds entropy to the cleartext.

[AudriusButkevicius]

I think the added complexity is not worth it, especially in the first iteration of the thing.

[ProactiveServices (Adam Piggott)]

Excuse the spam, would just like to thank all involved for the work on this feature. Very exciting!

[generalmanager]

@Lennix Have you had any more time to work on this? We had a very interesting discussion on Github about the technical details, including convergent and authenticated encryption schemes quiet some time ago:

Issue: Support for file encryption (e.g. non-trusted servers)

The proposed schemes would allow for trusted (plaintext) nodes and untrusted (encrypted) nodes to do deduplication, enabling massively improved transfer speeds and reduced traffic. As far as I can tell it’s impossible to tell if a block from an untrusted device already exists on a trusted device with your setup if you don’t want to transfer the whole file, decrypt and hash it first. This would basically limit untrusted devices to be dumb storage, only able to interact with trusted devices for a very high cost processing- and bandwidth-wise. With the convergent encryption scheme proposed by @bitshark in https://github.com/syncthing/syncthing/issues/109#issuecomment-77528026 untrusted devices can be fully functional members of a device group.

[u2060]

(Hi,) I’m really enthusiast about this feature too, especially since Wuala died and BT Sync’s implementation is still not perfect (cannot link devices & tell to automatically send only encrypted link to the untrusted node ; it sends RW links).

I don’t know if it can help for this, or for Syncthing in general, but Wuala’s creators published this paper (which was the origin of the software I suppose) : https://www.net.t-labs.tu-berlin.de/~stefan/srds06.pdf

[gmason]

Encrypted mode feature would be great!

[ny83729 (NY)]

I am sorry I can’t contribute to the technical aspects because I don’t have the needed security background. However, I wanted to highlight to Jakob, Audrius, and Lennix that this is one of the (if not the) most important missing features from Syncthing. For many users, the only way to make Syncthing sync files between office and home computers is to run a server as the always-on device. However, leaving the files on the server unencrypted is a huge security issue. Even a basic file encryption without metadata encryption seems better than what we have currently. I love Syncthing, but I feel very exposed with the unencrypted files on the server.

[generalmanager]

If you run your own server, you can probably also use encfs or ecryptfs locally (which is used by ubuntu to encrypt your home directory). I’m sure they know the importance of this feature, but +1 style answers only take time to read and delete (email notifiers), consuming time they could use to code stuff

[ny83729 (NY)]

encfs doesn’t work because someone who gets access to the server will get access to the files as well because to make Syncthing use the sync’ed folder, encfs would need to be automatically running on that folder.

I agree with you about the +1 style comments. That’s why I focused on showing how lack of this feature can become the biggest security issue instead of simply saying something like “I want this feature too.” I also understand that Jakob, Audrius etc are very generous to rest of us by providing this product. But, I’m sure they know that us asking for a feature does not mean we don’t appreciate their effort.

Ironically, this new comment of mine contributes even less to the actual bug than my previous comment. So, I’ll shut-up now.

[generalmanager]

encfs doesn’t work because someone who gets access to the server will get access to the files as well because to make Syncthing use the sync’ed folder, encfs would need to be automatically running on that folder.

That’s why I wrote “locally”. If only encrypted stuff leaves your devices, there are no problems. I’ve actually setup the encfs stuff for windows users and the ecryptfs stuff for linux. You simply choose the path of the encrypted files in syncthing. Doesn’t matter if it’s mounted somewhere, you can modify the mounted stuff and syncthing will sync the encrypted version.

[ny83729 (NY)]

Oh, great that you brought this point up because this may be a good option for Windows/Linux users! I actually was using encfs + Fuse with Dropbox on OSX. However, encfs + Fuse is very fragile on OSX. After losing a few files in the past few years with that setup, I lost my faith in it, and that’s when I switched to Syncthing. Removing encfs meant my files were going to be accessible to Dropbox. And that was unacceptable to me.

[generalmanager]

I also set it up on OS X for a friend, who hasn’t had problems with it yet (works for ~1.5 years). I can’t really remember the wrapper I used tough. It wasn’t plain fuse, because that didn’t really work.

[Lennix]

Didn’t have any time to work on this due to other projects demanding my time. I hope the code base didn’t change too much in the meantime.

@generalmanager

The proposed schemes would allow for trusted (plaintext) nodes and untrusted (encrypted) nodes to do deduplication, enabling massively improved transfer speeds and reduced traffic. As far as I can tell it’s impossible to tell if a block from an untrusted device already exists on a trusted device with your setup if you don’t want to transfer the whole file, decrypt and hash it first.

The untrusted device holds the hashes of the encrypted blocks in its database. So its able to determine if a block (of a file) has already been transferred. Deduplication of blocks is debatable. On one hand it saves space and improves transfer speed, but on the other hand it opens an attack vector. (Since you can’t effectively salt the block encryption)

[xerus (Jasper)]

Hi there, I’ll be joining in and see if I can help with this. I was actually planning a project similar to Boxcryptor but syncthing + encrypted nodes seems a far better idea