My first post in here so excuse me if I’m posting in the wrong category, I don’t really know exatly where to post this.
First of all I’m also one of thoes wanting to trash BtSync for good for letting me down as a fanatic user, or atleast letting alot of my fellow users down with the 10 folder cap, though the promised otherwise. So now I’m going to give this very apealling nice piece of opensource-work a spin and telling how I feel i regards to switching to this software, and hopefully this will sooner og later leat to me shifting for good and maybe I’ll put a donation your way for the good work.
Butr before I’ll go down this route, there are things I’ll wanna share (and hopefully get fixed) with your guys. This is not a how-do-we-make-syncthing-a-new-btsync-clone kinda thread, it’s more of a how-to-make-a-great-userexperience-for-former-btsync-peoples-making-them-want-to-stay-here-forever-and-maybe-donate kindof thread, if ya know what I mean.
This is a really great piece of work wich i’ll honor as best i can. Your guys are really moving the right way and you’ve done a lot off things so very correct. I especially love how the introducer-functionallity works and makes it easy to build a big mesh of device making syncing on a capable connection real fast. I’ll gladly rent some low-cost servers around the globe and set a lot of syncthing-nodes up so that my syncing across various platforms happens in the speed of light. I’ll love to work with that.
However there is one thing that really turns the show off and that’s the total lack of ability to påenetrate throug firewalls. Though i might easily do a port forward at home to reach my home-server there are a lot of networks out there that I have abolutely no control of and even the automatic UPnP fails. Mobile networks for example (3G/4G) does connect to the home-server sometimes but copletely refuses to connect to each other because UPnP does not work at all on them ant they’re basically NAT-networks at most. This downs the transfer speed a lot when nodes needs to coonect to my home-network with low upload speed. This is really what makes BtSync awsome and great. You can just exchange the secret and you’re sure to be connected anytime you connect to the internet around the world. It just start to sync and works very well. And you should really walk down that route. If we can have this functionallity in Syncthing, if it kjust will find a way to work (maybe thru relays) Syncthing is will be so f****ng awesome.
I know this has been discussed before but I really think this is what is missing right now. You should really work on some relay functionallity og maybe even let every instance of the Syncthing function as a relay in the way that Skype operates. Every instance of the Skype-software is a relay too so encrypted trafic can easily pass around the internet and get thru everywhere.
This is my biggest wish for this project. As soon as I can rely on Syncthing to just work across any network I’ll gladly jump of the BtSync-train and make the shift and put Syncthing into my day instead. This is really what is holding me back from using it for other than te3sting right now. With Btsync i can rely on it to work everywhere on any network reliable, with Syncthing on the other hand i can’t because I need to make sure that the port forwarding is in place and UPnP i very unstable at best. Sometimes it will connect across NATed networks bnut offent it won’t and I’m ending up using more time on messing around with router settings and other settings to get it to work, wich is a big shame.
What will it take and why doesn’t we just jump aboard? If the main problem is privaci then make it a choice for the users to deside to turn the functionallity on or off in the settings and then make som relays awailable or even better make every instance of Syncting a relay to cover any area of the world. In this way the speed can also be provided by spreading the packages around the relays.
Yeah, so there are multiple ways to solve that problem as you say.
One is UDP the other one is relays. UDP being the difficult solution is in the progress of being solved by our brilliant @calmh. Relays is the easy solution
I would just like to say that I am very happy and impressed with this. I am able to deploy and manage folders remotely with zero user interaction (other than to uninstall BitTorrent Sync), and with the help of NSSM, I can always be sure things are running. Kudos!
I had issues with discovery over static VPN routes, but really only needed each remote node to talk to a main node anyhow, so it was nothing to add the IP manually. SSL certs needed help from openssl to convert to pem since it’s a Windows only shop here, but had no issues with HTTPS at all!
@AudriusButkevicius That’s definitely something worth looking forward for…But why not go all in and use both solutions. That will indeed ensure connectivity for everyone but the most cribled network setups?
@stewy Creating a star scheme where every node talks to the same one and only node is no problem but if you want to do a mash scheme where every note talks to each other then it’s a hell lot more complicated. The good thing about a star scheme is the simplicity over speed but if you want alot of speed and a sync you can rely on even if a node og 10 goes offline for some reason then mesh is the route to go.
Great comments. I really like your candor LinuxBaby. I agree with what you’ve expressed. Honesty and frankness is what enables Syncthing to grow and improve, and it improves (speaking for myself) my own talent, expertise, and understanding.
There is alot of potential in competing with BitSync, and I think Syncthing may displace soome of it’s user base without even trying, but there is also the issue of coordination and prioritizing what we are all working on. But I think that will all work itself out, as there are some good minds here.
But managing the features and workload aside (I have no idea how the core team did this in a year, it’s astonishing)… Anyway but in regards to the problem of “Bypassing NAT” and other features, enhancements, etc – I think Audrius and the other main devs are well aware of these issues – including the benefits and drawbacks etc. It’s a balancing act.
So Audrius and I have gone back and forth on the transport protocol, TCP vs UDP, that and other issues, because any major changes to the client or protocol could turn into a huge project. Yeah, my experiments show UDT is a bit faster in experimental conditions, but if it takes a year to implement, is it worth the trouble? Tough questions.
Anyway, everything in Syncthing is TCP now because of a few reasons… I agree with Audrius’s inclination to stick to TCP in the short-term …
Benefits ot TCP for short-term:
Way easier to debug than UDP
Ensures ‘built in’ packet ordering, session management, congestion avoidance, ACKs of data, etc.
TCP and TLS work great together, and means we don’t have to deal with rolling up custom crypto or UDP TLS
TCP will scale well enough for small network (say up to maybe a dozen or two dozen nodes)
So TCP will be fine to get the concept perfected.
The big disadvantage , however, for TCP is exactly what you’ve described in the original post… If I have one Syncthing endpoint behind NAT, and a second Syncthing endpoint in another city behind NAT, how can they establish a TCP connection directly to one another ?
Peer A ←→ Gateway A (NAT-a) ← … Network … → Gateway B (NAT-b) ←→ Peer B
I might be wrong or misunderstanding the issue, but I don’t think they can directly connect to one another in the current implementation. At least not using standard means (they can’t just open a TCP socket to one another).
This brings us to the topic you’ve brought up … the concept of bypassing the two-sided NAT … What are the ways to do this?
Audrius pretty much summed up the options. Here they are in detail…
(1) Relay Server.
Both NAT’d hosts connect to a relay server (usually via TCP), and the relay acts as an intermediary. All traffic flows through the relay.
Peer A ←→ Gateway A (NAT-a) ←→ Interwebz ←→ Relay ←→ Interwebz ←→ Gateway B (NAT-b) ←→ Peer B
The downside is someone has to pay for the relay server . And bandwidth can be expensive . There is no way we could afford to host this on Amazon . It’d cost a fortune. Even a VPS or dedicated server might be too pricey.
This IS be a possible solution for how SyncThing could afford a Relay Server for all users, but we’d have to find a low-priced colocation with unlimited inbound and outbound traffic for a reasonable price. Kind of like the Syncthing discovery server that’s already running, this colocation machine would act as the ‘Official Syncthing Relay’ – as a public service for anyone behind NAT.
(2) TCP Hole Punching.
This is hard to describe, but is pretty much a complex technique of tricking the NAT/ Firewall into thinking both endpoints are establishing outbound TCP connections, but really it opens a temporary ‘gap’ in the firewall so that the two peers may rendevouz and establish a TCP session.
This involves port prediction and/or tricky manipulation of raw sockets (manipulation of the TCP handshake to trick the firewall with SYN flags). Usually there’s a limitation here , because to manipulate raw sockets (send crafty firewall-busting TCP packets that deviate from the RFCs) you need admin / superuser privs.
This technique is difficult and not 100% reliable. And I think it’s used in Bittorrent . It might need root / admin depending on the method. But it might be an option . Perhaps others can weigh in on TCP hole punching.
(3) UDP Hole Punching.
Similar idea to TCP hole punching … Several different methods of doing it . If I recall, most involve a third party rendevouz server that helps the NAT’d parties agree on the details of their UDP packets – mainly which ports the endpoints are going to send and receive on. The rendevouz server also helps the NAT’d boxes open up the holes in the firewall.
The nice part about UDP hole punching is that traffic flows directly from Client to Client once the rendevouz server has helped them punch the holes. It doesn’t require all traffic to flow thru a relay … Just the first few packets need the third-party server. This technique is WIDELY deployed in P2P protocols like Bitorrent.
Now remember UDP is ruled out for the moment. I think Audrius mentioned a developer was adding this in a feature , but it could take a bit to get finished, tested, integrated, etc etc.
So that leaves us two options to connect two peers both behind NAT, in the short-term (<12 weeks)… We can
(1) Set up our own Syncthing relay server .
Not a VPS – A real 1U colo on a high-speed datacenter somewhere in U.S… This would be the public relay for any connections. Peer1 connects to relay on a control port, and is given a random port like 31000. It reconnects on the designated port. Peer2 does the same, and is issued a different random port like 31001.
The server uses iptables, netcat, a reverse proxy, custom code, whatever, and bridges the traffic between relay:31000 and relay 31001.
We’d just need to make sure the relay server only allows Syncthing , and not other random traffic. And then NAT is not a problem any more.
For the NAT-busting features discussed here, I’d say catch up with the main devs and see what they’ve got going.
What I’ve written here may already be possible. I’m not clear on what parts of this are completed, planned, or on hold.
But if I were operating ‘from scratch’, I’d say let’s make a Relay Server. I know I have a lot of ideas – but this is actually relaly easy. I already have two colos, a half dozen VPS, and at least a dozen EC2 instances.
So here’s how it’s done… We get a 1U server – something used. Maybe like an AMD multi-core with a ton of RAM. We either set up RAID0 on a couple of small SCSI drives (40GB tops) , or else we put in a couple of name-brand SSDs.
We install either OpenBSD or a solid Debian-derived Linux distro on this 1U rack server. We set it up with dm-crypt and crypsetup with full-disk encryption (probably like aes-xts-plain64or aes-cbc-essiv:sha256), with ramdisk for /tmp and encrypted swap partition. We can make it extra-hardened with grsecurity kernel patch
This Relay Server would be the rendevous point for any Syncthing client that can’t find a peer because of NAT – The Relay routes which was registered with announce.syncthing.net, but where two peers attempted to talk (and failed) because they are behind NAT / firewall. The relay would accept two incoming connections, do some sort of validation, then we just bridge them (relaying traffic between them).
This could easily be done on the Relay Server via ssh, iptables, custom solution , or some combination. Technically the ‘relay’ is a bit like a reverse proxy. Here are some good details on how it’s done…
Bridge between two ports on the same network interface on a Linux server
But how do we afford to pay for a dedicated colo? Easy. Cheaper than you think.
The cost of hosting a 1U colo would be around $60 a month for unlimited bandwidth at a Chicago datacenter.
–^ example of $60 a month colo for 1U rack with unlimited bandwidth… peering point is near Chicago, which is a pretty good place to host a colo for continental U.S.
OR… we can…
(2) Implement TCP Hole Punching into the codebase.
This sounds like a major headache… Mainly because I don’t know how to do it. But maybe someone here does know how to do it – or maybe there’s libararies we can use to accomplish this.
Those are my thoughts.
OH last hing: The Network should absolutely be fully meshed at small scales – unless there are a large number of peers. But up to say maybe like 6 to 18 nodes, I strongly believe the network should be fully meshed (or close to it).
For larger networks it’s not going to be possible to go fully meshed because of the overhead of TCP , especially if things get chatty. You might be able to get up to 40machines mostly fully meshed if you switch over to UDP, but that ain’t gonna be pretty. And UDP is not going to be feasible in the short term.
For larger networks you’d have to copy a sort of partially meshed style, with distribued structure. There’s a reason bittorrent isn’t fully meshed – haha… It’s not possible. All the traffic would grind to a halt.
That’s because the number of connections (aka total TCP sockets) between nodes in a fully meshed network grows quadratically with the number of machines in the network. In Big O notation, the number of TCP connections grows relative to the number of machines as O(n^2).
Specifically
Let c be number of total connections in the network (graph edges)
Let n be the number of nodes in the network (machines in fully meshed net)
So fully meshed is really convenient , nice for small nets, but is awful at scaling.
Sorry, I mean the NAT punchthrough thing, like bittorrent, I want to sync my files to home whenever i wanted regardless whatever my network condition i have. I know something like ngrok can archive my goal, It would be nice if syncthing integrated ngrok’s ability into syncthing. ngrok is written in go though.
Sorry for being late to the party, but I wanted to point out that I can’t imagine a way this could be satisfied without breaking the peer-to-peer security of Syncthing. To the outside world, Syncthing connections just look like TLS.
In order to enforce this you would need to look at the data being exchanged inside of the TLS channel and ensure that it is in fact the Syncthing protocol. And if you can do that, you can spy on all of the data being exchanged and I doubt that will fly.
False. Before relaying is started they have to ask relay server to allocate a session key (via command socket) then they need to send the session key as the first message to the relay socket before the two parties are connected to each other, which then do the tls handshake etc.
If you go the extra mile to reimplement syncthings relaying protocol on top of your thi g, then I guess you can feel free to use the relay servers.
That doesn’t really contradict anything I said. In order to enforce that the peers are talking Syncthing to each other you need to be able to inspect unencrypted data flowing over the peer connection.
I don’t really want to resurrect the above five month old thing, but I just want to add that yeah, the relay servers just bounce packets (after the handshake described above) - I don’t really care much what kind of packets gets bounced.
Thats one decision you’ll regret one day when you pick up on the news just to learn that some big company had a DDoS attack and that the origin seems to be your announce servers IP. that will definitely not be cool.
However there are actually ways to get over this problem pratically speaking, I’m not a programmer of and definitely not a GO-programmer but I have some ideas you maybe could use.
If you have security in mind you could look at the N2N-softwares take on security.Let me just shortly explain how N2N works
N2N is basically a peer2peer VPN system where you have the binaries splitted up in two. A supernode and a “edge”. The supernote is meant to be run on a computer which is visible on the internet. Lets say you have a computer with port 1234 open on a statically IP-adress. When supernode is started it basically just becomes a big “phonebook” for every nodes connecting. The “edge”-part of the binaries is then started with a community-argument and a password. TRhe community can be whatever word you like as long as every edge-clients uses the same community. when every single nodes usese the same “community” and the same password thru the same supernode they can contact each other and form a network which they can communicate freely on. The supernode itself wil not decrypt the community, the onely thing it does is to let any number of nodes communicate with each other. The supernode kan i fact handles any number of communities completely transparant from each other. In fact there even exists public supernodes that you can freely use as you like without the risk of anyone eavesdropping on the communication. The supernode itself will accept any combination of community-word and password that you can think of but only if two nodes connects with the 100% exact pair of community-word and password they will be able to communicate with each others. If the community-words differs or if the password differs the clients will not see each other at all.
So how could this be implented into syncthing.
Well evry instance of the syncthing server vill communicate thru the announce-server and will have a randomly generated community-name in its settings that the owner can edit at any time. Having the same community-string and password set on different syncthing-instanses will let them communicate with each other freely. The user will still need to exchange the usual keys but the servers will be able to see each other then the keys and the community-string/password is equal. If the community settings does not match any connected clients then the crypto will be wrong and the server vil just send out fake TLS-packets which will fool the client to believe that it had connected successdfully to the server with zero other clients that it could connect to
So every login will yeld success as seen from the client side but if the credential is not right the client just cannot connect to any other clients as they’re on another network seperated from each others. The supernode/Announce-server vil just serve TLS-packets to the “community” and noone else, and it will not at anytime decrypt the packets. It just relays the packets to the whole community. The user could at anytime edit the community and create another community should they desire that. As long as all users are on the same community they can communicate with each other.
No. The relay servers are rate limited and do not amplify traffic in any way. They would require the target to connect and request a session, too. They are completely useless for a DDOS. Please stop baseless speculation without even looking into the technical details.
What about key-based ssh-tunneling? No need to reinvent a wheel, keeps it secure, and exchanging the key would ensure each system was “owned” (thus secure, distributed, etc - all of the reasons for using this type of system) by the same entity.
Seems like that would solve the problem with firewalled/NAT <—> firewalled/NAT systems having difficulty communicating. Unless I’m misunderstanding the main point of this discussion (which is definitely possible).
I suppose it requires clients (Android comes to mind) to have ssh access which they may not have access to, similar on a standard Windows system… Yeah, good point. Just kinda thinking out loud and apparently not helping
Also you’d still need an intermediate point to send the SSH traffic to in order to setup communication. I was just envisioning reverse tunneling:
System1 | NAT | Firewall <----> Firewall | NAT | System2
If I want to access System1, I can setup an SSH server on System2 and ssh to System2 from System1 to create the reverse tunnel. However, I have to port-forward System2 on the firewall to allow System1 to connect, so that’s not a solution, at least not directly.
