Here’s to give some heads up on coming changes. I’ve been hacking on something I’ve called the Datagram Stream Transfer protocol - it’s basically yet another reliability layer on top of UDP. There weren’t any existing solutions in Go that I could find, I didn’t want to go with straight uTP, and the UDT protocol spec is both confusing/incomplete and not perfectly suitable for our purposes, so I made a little mishmash. It’s mostly based on UDT, but differs in quite a few aspects and is also not finalized or specified in a formal way yet. I intend to do so before it’s “deployed”, so it can be used by others or reimplemented as appropriate.
Here’s what I envision for the next few upcoming major-ish releases;
Add DST support and use it by default. Also listen on a regular TCP socket and fall back to that in case DST breaks or the user configures it that way.
Add DST support to the discovery server. Announcements will then be made as a TLS-over-DST connection, thus proving the announcing devices identity to the server.
At some point, probably, deprecate TCP.
Point number two (using DST for the discovery protocol) is somewhat crucial, because it enables much better NAT busting. Basically, as long as we use the same UDP socket for BEP and discovery it doesn’t really matter if we can get a port forward with UPnP or not - the discovery server will see the real port number on the outside, and this port is what is to be used when contacting the device from the outside. If the NAT device does full cone NAT, that’s all there is to it. If it doesn’t, you can anyway get a connection in most cases by doing a simultaneous connect from both sides to the other.
It also gains us secured announcements, although the answers to discovery queries must still be assumed to be potentially forged. We could also do regular, centralized HTTPS-style certificate verification of the discovery server if we wanted to.
Also, we should really get it fixed so that announcing to and querying multiple discovery servers work. This is necessary both for real IPv6 support, and for some interoperability now that Pulse is out there and Ind.ie has gone it’s own way with another “global” discovery server (summoning @aral, @GeorgeMac here).
I am very excited about this one.
While we are at this heavy change I’d like to suggest a few more modifications to the protocol since we are in this transition period (not sure if any of these break backwards compatibility though):
Add block hash to requests so we could do hash based look ups rather than file/offset/size in the future.
Add options (same as cluster has) to ClusterConfig->Folder structures, these can then be used to carry custom information about the folder which future versions might support (such as folder labels we are talking about, encryption keys or whatever else)
Alternatively, I’d be in favor of moving into some more flexible protocol definitions where it wouldn’t be as painful to add and remove stuff, but that’s a lot of work which is not going to yield any benefit in the short term.
Now discovery… I started working on new discovery twice now.
I have a basic DNSSEC + TLS/TCP based approach skeleton but I’ve realized that it will not work because of multiple issues…
It could still be done using DNS (which helps carry the load), it would just be a bit more hacky.
I decided that as a very basic first step I’ll move the announcements to TLS so we could prevent DoS.
I realized that there a few other problems before that, and started hacking through the UPnP, announce and discovery code supporting multiple UPnP mappings, multiple announce servers and so on…
I thought that once that is hacked out, I’ll move the TCP announce part to TLS and add the verification of the announcer, but given it’s needed I can move querying to HTTPS too.
I guess DST announcements would still have to be done via a separate socket, so my changes have very little to do with this, and given we do move to HTTPS querying, we still have to support old UDP querying/announcements (which will have no way of announcing DST ports) I’ll have the opportunity to have DST compatibility in mind.
I didn’t really want to go this route before, for a couple of reasons. The overhead of setting up a TLS connection just for a query is quite significant, and could be used to DoS the discovery server quite easily. Working around that by having a persistent connection per client takes up too many sockets on the server and doesn’t really scale.
However, DST sockets are all multiplexed over a single UDP socket, so only uses resources in the actual program to keep track of, so long lived connections are fine. Hence I was thinking of letting each client establish a persistent DST connection and keep it open with a keep alive packet every few minutes. That also gives the server a better indication of who is still alive out there. From the network stack side of things, this is just a single UDP roundtrip, exactly as it happens today, except the packets are encrypted and actually part of a TLS session. Queries can be made over the same connection.
From a code perspective, this all works exactly as if it were TCP, except one does a mux.Dial("dst", ...) instead of net.Dial("tcp", ...) to get the connection, so any code that uses TCP+TLS should work pretty much out of the box with a few trivial changes. The mux wraps an existing UDP socket and can be passed around among packages that need to use it to establish connections.
Yeah maybe, or the new version is not compatible with the old.
Hi Jakob, thanks for the heads-up. I can’t really oversee yet what the benefits and disadvantages of DST over TCP will be. But it sure sounds like an interesting experiment.
I’m also very curious to see how well this would work in a mobile (Android/iOS) environment.
For very selfish reasons, I hope that you will keep the TCP support for a while, because that will make my job of porting the protocol to Swift and Java a bit easier
I guess first things first.
I’ll add support for multiple upnp mappings (as they are staying for now atleast), announce/query servers, add a Protocol field to the Address structure, add a migration to bump the port and then MAYBE move stuff to TLS which gets rid of part of the problem and closer to TLS over DST, and we’ll take it from there.
I agree your idea sounds very cool, and will yield much better discovery results, but I still think it is still vulnerable as it still has to go through the heavy client hello business due to TLS.
And given I am evil, I can just keep opening and closing TLS sessions until you run out of memory, given I do it fast enough.
The major one should be better NAT penetration without the cooperation of the NAT device, hopefully without being much slower and crappier than TCP. Anything else is just an extra bonus.
Yeah, I don’t think this will hurt. In the end we won’t really care if we’re running on top of DST or TCP so can keep trying them round robin until one or the other succeeds.
This is fairly easy to rate limit though, much like we do already. Of course I’m sure we won’t be invulnerable (a simple DDoS will take anyone out of business), but not entirely defenseless either. :)
