We should brainstorm and work on some cool new features based on what people generally want… then it will become better.
Anyway, I may start doing bandwidth optimization, but it seems to make most sense to me there to do the reference implementation in Java or C#/C++ first. Changing the data structures of of the initialization handshake would help… But this is hard stuff. This is why I think it’s so cool.
I think we need to put in our own sort of “DHT” of the files on the system that can be easily serialized to disk and to the network. We also need to have the state of the files on the local machine shared with the other machines, which triggers the update as mentioned.
Also perhaps we store the same descriptors of one’s peers as well. So you have LocalState_Machine1, LocalState_Machine2, as objects. GlobalState is the union of these objects.
My personal inclination is this… Store the descriptor of the filesystem you are mirroring locally in RAM, a RAMdisk, a networked encrypted cache, a local disk cache, distributed hash table, whatever – store it OUTSIDE the FS we are monitoring – and also store it internally as an object while the program is running. Save this serialized object on program close to disk regularly in serialized form.
This way, on startup, you’ve got ‘where we left off’ in terms of who has what (files) before we even begin transmitting packets!
FASTEST PROTOCOLS TO TRANSMIT DATA?
UDP
UDT
TSUNAMI UDP
UFTP
FDT
So we need to ditch TCP if we can avoid it. It works fine with one stream, but if we want multiple parallel streams it’s way too much overhead in terms of latency etc. UDP with a checksum and an ACK is fine for our purposes.
If the packet fails, we can simply move that chunk to the top of the stack ad request a retransmit. There are even libraries tha will do all this for you – monitor throttling and so forth for UDP. Oh and UDP can take multiple paths thru different routers so it’s inherently parallell.
Another HUGE issue is we need to change the way data is exchanged My personal opinion is that we need to switch over to UDP, but that is a big undertaking. In fact, I’d switch to UDP (like bittorrent is using) , ditch the TLS entirely, use an open source flow-control library, and then roll my own encryption using EC-DSA and not RSA.
But I know there is this animosity towards bittorent here, but my take wouuld be ‘yeah fine, let’s write a better protocol’ , but I think it’d be really neat if I could sync all my files to my machines and encrypted to Google Drive… Then with one click, I can select an album I recorded and publish it to bittorrent by the DHT system.
I really think we ought to take advantage of existing technologies as long as they are decent… A distributed hash table is good idea, especially if people want to share data outside their own clusters … ie giving someone on facebook a link to your data store, etc.
But that’s up for discussion and would be long term. Short term, for speed increase (ad the ‘bittorrent-like’ ablity for the network to speed up as you add more machines)…
We need three big things for that to happen…
- UDP Protocol for sending data chunks. Multiple parallel streams. Up to 10-30 pckets ‘in transit’ at once, easy.
think we can learn alot here SRTP protocool…
-
REDUCE UDP packet sizes. The max sze of UDP packets is 64k anyway. Otherwise they will just get fragmented on the Tier1 backbone … Also select UDP pack size based on real world testing.
-
Use a UDP library designed for dealing with congestion , so it throttles up and down. We can still use TCP as the negotiation channel for TLS if we want, but I think the packets should be UDP if possible.
4… Multithreading… Not only the GU needs t’s own thread… Or the socket code… The app will need tons of threads. All network I/O should be non-blocking, and instead should fire off when data is received or at predetermined intervals. In terms of writing data to files on the disk, we definitely need to consider always calling flush() and fsync() in a nonblocking manner as well.
-
Utilize in-memory disk model as a hash table… Check disk for changes and if so, update the hash table.
-
Current ‘version’ of files should NOT just be Folder and Name. At the very least it should also include the length and some sort of hash. It can be something like File_version = SHA2_256(File_Name + File_Data + Folder_Name + Lamport_Clock).
7… Anytime a host confirms reception of a block, immediately share it with authorized peers if they don’t already hve that block. Share the block in parallel (threaded), and do it PRIOR to flushing the block out to the disk.
- Trap filesystem events at the kernel level if possible.
Minimaly, we need to lower the ‘chunk’ size dramatically, but have it scale with the file. Chunk size is not as much of a factor over TCP, but it’s critical for UDP. Both are important because of network MTU , which varies, causing IP fragmentation in transit over WAN.
Files up of size up to about 128k or so should be transferred all at once… Especially if they re part of the protocol control metadata.
But larger files MUST have the chunk size lowered, when we get into GB. And we need to draw inspiration from the bittorent block mechanism, where each machine know what files it must find missing blocks for.
For starters, we can just have the UDP threads grab missing blocks… But ideally, we want the hosts to exchange information on the rarity of a particular file’s individual block… Then they should be fetching the rarest blocks first.
Also, each block needs to have it’s own (AT LEAST) message digest like SHA128 or whatever , so we know if it arrived intact and untampered.
Preferably, each block has an HMAC (extremely fast and secue) which we derived during decryption o f the data and stored in a local hash table or RAMdisk.
Having the hashes of all the file chunks – expected vs reality – will safe immense amounts of time, and will prevent corrupted files. If a block gets corrupted, we can autodetect and have only that block retransmitted. We rescan the file, and then we can say we’re in sync (rather than doing full gui rescan all the time).
So less filesystem monitoring and more asynchronous parallel multithreading.
Anyway what I’ve written here is alot of work I know, and I don’t propose to say someone else do it, because it’s not easy. But I’m confident this will speed up the transfer speeds dramatically.
By the way, are any of the active developers here located in San Francisco? I’d also like to chat over email, Skype, or the phone with any of the people who’ve contributed to this project.
Again, great job, and amazing work.
-Alex
… that’s a long list of “bugs” and Wikipedia says