Syncthing sync (not management GUI) behind HAProxy - using SNI

Hi Everybody,

Thank you for Syncthing and this amazing community. :wink:

I have a webserver with multiple websites in multiple containers and with HAProxy that terminates SSL. In other words a client hits the server with Server Name Indication (SNI) myweb1.com in HTTPS request and the HAProxy decrypts HTTPS and send the HTTP traffic to a container that hosts myweb1.com. And same for myweb2.com, myweb3.com

I’d wanted to repurpose the server also to sync data. I managed to install Syncthing in another container. I got Let’s Encrypt cert for syncadmin.mydomain.com and I route traffic from this subdomain to Syncthing GUI management. Another subdomain sync.mydomain.com is pointing to the server and I can use iptables/nftables to route traffic from unused port (by default 22000) to the container with Syncthing.

In other words if I connect Syncthing GUI management, the traffic goes through HAProxy. Data synchronization bypasses HAProxy and goes directly to the container.

And it’s working great!

One thing I learned is that with this configuration I don’t need to use Global Discovery Server. The Syncthing instance in container can disable Global Discovery and clients can point the server directly by specifying: tcp://sync.mydomain.com:22000 in Advanced tab when adding new remote.

I was thinking about running multiple Syncthing instances in multiple containers. One for me and other completely isolated for my family.

I wanted to try to run it through HAProxy and I made the following configuration to work:

frontend fe-tcp
        mode tcp
        bind :22000

        default_backend tcp-syncthing1

mode tcp in HAProxy config tells HAProxy to pass-through the traffic to the backend (container) without checking SSL certificate. Opposite mode would be mode http that I use for all other services since all other services are just web servers (including Synthing management GUI).

Another level (that I can’t make it to work) would be something like this:

frontend fe-tcp
        mode tcp
        bind :22000

        use_backend tcp-syncthing1 if { req_ssl_sni -i sync.mydomain.com }
        use_backend tcp-syncthing2 if { req_ssl_sni -i sync.otherdomain.com }

But that doesn’t work. In other words Syncthing doesn’t (I guess) use SNI.

Could you please tell me is there a way to differentiate multiple client requests? Do clients include for example remote domain name or Device ID somewhere in the request?

P.S. I don’t really need to make it work since I can simple use port 22000 for the first Syncthing instance, port 22001 for another…, but I’d like to try to make it work over a single port as a kind of a research project ;-). Thank you.

Kind regards,

Matej

I don’t think Syncthing sends anything in the handshake that will help you determine the intended recipient device, certainly not SNI. It will however retry if it went wrong, so in theory you can 50/50 load balance it and it should work eventually. :slight_smile:

1 Like

Just some random thoughts of mine, not actual solutions

  • You can differentiate between HTTPS and syncthing traffic using ALPN, but this doesn’t help in differentiating between two syncthing connections (Syncting on Port 443 behind SSLH - configuration).
  • Syncthing’s default certificate common name is “syncthing”. You could in theory use your own certificates with custom common names. But this requires quite some amount of configuration. Edit: This has the same issue(s) as the third point.
  • Device ID is based on hash of the certificate. But this isn’t helpful when determining the destination of a syncthing connection (in TLS the server needs to send it’s certificate first).
2 Likes

Thank you very much @calmh and @Nummer378. I was afraid that I’ll get an answer “stop hacking into Syncthing, go away”. Thank you.

Interesting idea. Thank you.

Thank you @Nummer378. You’re reading my mind since I’d love to run Syncthing on 443 with other websites. But that’s phase 2. Phase 1 is to run multiple Syncthing instances on a single unused port.

Ouh! I had no idea! I googled this article how to replace the key.pem and cert.pem? where @calmh explained:

So you can pre-create the directory and certificate+key, and on first startup Syncthing will generate the default config etc.

So yeah! I’ll dust off my openssl skills and try it. Thank you both!

1 Like

I had more time to think about that and I realized I can’t use Common Name to differentiate different Syncthing instances.

Even if I replace certificates of Syncthing instances behing HAProxy with certificates with custom CN (for example mysync1, mysync2, mysync3) and I specify certName mysync1 in Syncthing on my PC that I want to connect to mysync1 container, my PC won’t specify mysync1 in the request. This is what SNI is for and as @calmh mentioned, Syncthing doesn’t send it.

Syncthing doesn’t send anything interesting in the handshake and therefore there is no way to determine on HAProxy where the traffic should be routed to. On HAProxy I might be able to read the my PC cert CN (I’m not sure about that) but not the destination (container) cert CN.

Yeah I guess you need to handle this at the TCP/IP level via different ip addresses or port numbers. Or randomly guessing (balancing) as calmh suggested, though this will cause some handshake failures.

1 Like

Load balancing is a clever idea but it’s also a “dirty” solution :wink: . I think I’ll just use alternative ports or if I really need to use 443 (other ports are blocked) I’ll just request some public IPs…

Thank you very much both!

1 Like

Going to take the opportunity to mention that with IPv6, not having enough IP addresses available isn’t an issue :grinning:.

I know that like 2/3 of the world hasn’t heard of this magic thing which has only been introduced 24 years ago (almost 25 actually), which is why I encourage admins to (finally) deploy it wherever possible.

And I can get /64 free of charge from my VPS provider. The problem is that my ISP has no clue about IPv6… Of course I dial a VPN over IPv4 with dualstack endpoint but that means that everybody including my mom who wants to use Syncthing would have to do it. Not a great solution…

1 Like

And I thought I provided high-levelish family-infrastructure-support - apparently mid-level at best :smiley:

We could send the user configured device name or the expected device ID in SNI. I mean, there’s no reason we couldn’t, but maybe there are reasons we shouldn’t, and only niche reasons for why we should…