Multithreaded encryption and transfer

Support
1

Syncthing is becoming a pretty nice alternative to closed source products, such as BTSync.

One of the things would be great to see is multithreading in encryption and transfer.

Encryption is CPU intensive and while it is going on, the transfer can not happen in a single threaded design.

At the same time, network transfers are not CPU intensive and could run at the same time while encryption is going on.

If you create separate queues for encryption and transfer, then you could simply place an item on the completed queue for encryption and place it on the transfer queue.

Also, it would be great to see several log files, such as transfers, errors and various status messages as to the progress of things.

Also, seeing some real time message in GUI as to current file being transferred and its progress in percentage of total size and total file size would be extremely useful in order to gauge some things of dynamic nature.

It would also be nice not to see various error messages, such as “unexpected EOF” (this one seems to have been taken care of in 0.8.17) or ping timeouts and other things that are not indicative of a dead state and inability to progress. It seems like the way it is right now with title bar colors and status display messages in GUI is sufficient. About the only error or status messages that one would probably like to see is total inability to do further processing. Anything else simply takes away attention of the users from important things.

Another thing I noticed, and I hope I am not missing something, it seems like syncthing GUI is loading the CPU pretty heavily on the Linux server. When you kill that syncthing window in web browser, the CPU useage is dropping to normal. (I am using the https version with password to access the server). That means you have to keep killing the window and opening it up again when you want to see things.

Anyway, it looks like developers are working hard and non stop on improving things and adding power and features to the program and they do deserve and applause in a true sense of the world. Such dedication will certainly bring results and be extremely useful for many people “out there”.

Keep up a good work and keep in mind that you are not alone and many people do appreciate your work and find it valuable.

1 Like
2

Hi! Thanks for the writeup - there’s a bunch of things in there that should be addressed, and some that are at the top of the todo list for 0.9 (i.e. the GUI does put too much CPU load on syncthing). I’m travelling at the moment with sporadic internet access and just about zero syncthing time, but I’ll get back to this during next week. :slight_smile:

3

I don’t think this would gain us as much as you think it would. The protocol stack as it looks today is

  1. BEP
  2. Compression
  3. Buffering
  4. TLS (Encryption)
  5. TCP/IP
  6. lower layers

Data for a full message is written in chunks from the BEP layer, compressed and then caught by the buffering layer. When the compressed data exceeds 4 KiB or the message is finished, the block is flushed down to the TLS layer. Repeat until the message is finished, when there is a flush call that ensures all data is flushed down to the TLS layer, if there is a small block remaining. TLS passes the data on to the system TCP/IP stack. Layers 1-4 are inside syncthing and essentially runs on one thread per connected peer. The TLS layer is from the standard library and does all TLSy things - framing, authentication, hashing, encryption, etc.

Things will change slightly in v0.9 where larger blocks will be written already at the BEP layer. This will increase efficiency somewhat in that there will be fewer memory allocations, less back and forth with small data blocks, and data can be more efficiently split into segments at the TCP layer. I don’t expect this to be a measurable improvement though.

Theoretically we could separate things to separate threads for compression and encryption, but the compression isn’t that expensive really. The transmission part is already handled by the OS so is done in parallel with compression and encryption on a separate queue, after data has been handed to the TCP/IP layer.

The encryption itself (i.e. the TLS layer) can’t be parallelized for a single peer because it’s a stream cipher (each piece of data is dependent on the preceding piece).

Could you be more specific about what you’re missing, from the console side of things?

This would be nice. I’ll keep it in mind. It relates to …

… which is entirely true. Currently the GUI polls quite insistently (with some caching, but still) - “how many files are there in the repo?” synchting goes to count “how much is missing?” syncthing goes to count etc every ten seconds. I intend to replace this with an event driven GUI instead, where the GUI only subscribes to events and does nothing when nothing is happening. Look for this somewhere along the v0.9 train, but probably not in v0.9.0 directly.