Discovery server setup and client certificates

Support
1

Hi, I’m attempting to setup and use a private discovery server.

The discovery server:

  • Has a proper (private-)CA-signed certificate.
  • Runs on HTTP, behind Haproxy.
  • In the proxy, I’m appending the X-SSL-Cert, X-Forwarded-For, and X-Client-Port headers.

But it appears as if syncthing is not sending any client certificate (determined by monitoring the proxy’s output). If I send one manually with curl, the “no certificates” message on the discovery server’s debug log ceases appearing. As far as I can tell, the server runs fine.

Is there a special way that I need to declare the server’s url in the syncthing settings? I have it as https://stds.home --is the format correct?

Any insights? Or code location for the discovery process and the client certificate attachment?

Thank you.

2

IIRC, the server needs to request the client certificate.

1 Like
3

Last time I checked, syncthing clients do not always send certificates when talking to a discovery server. They do when they have the intention of announcing themselves, but do not do this when just requesting data (about someone else).

For the discovery server, this means that the TLS terminating proxy must always request a certificate (otherwise the client can’t send one), but must also proceed if none was send by the client. Additionally, the certificate must not be verified regarding trust, just passed through to the backend via the documented header method. In nginx this option is called optional_no_ca, which makes certificates a) optional and b) disables trust checking.

1 Like
4

So I have haproxy configured in a similar way, but I don’t know if is exactly equivalent to nginx’s optional_no_ca. In the bind line I have:

verify optional ca-file <ca> crt-ignore-err all

Haproxy’s verify documentation: HAProxy version 2.0.22 - Configuration Manual

“If set to ‘none’, client certificate is not requested. This is the default. In other cases, a client certificate is requested.”

I’m don’t know much about client certificates and how they are offered. Maybe it can be requested more explicitly? Or may there be a bug in Haproxy? How would I go about troubleshooting the transaction?

On a side curiosity-motivated question, since the client certificate is not verified during the SSL handshake, is there reason why it couldn’t be sent in the POST payload?

5

It’s not verified to be signed by a trusted CA, but the TLS handshake still proves the client has the corresponding secret key.

1 Like
6

Based on haproxy’s documentation, you’re requesting a client certificate.

You could try using ca-ignore-err all instead of crt-ignore-err, to see if that changes anything.

How are you forwarding the client certificate in Haproxy?

7

ca-ignore-err leads to an SSL client certificate not trusted error on haproxy. It is not useful.

I have something like this in haproxy’s configuration (in the frontend section):

http-request add-header X-SSL-Cert %[lua.ssl_c_pem_escaped] if { hdr(host) -i stds.home }

ssl_c_pem_escaped is from here: https://gist.github.com/gkatev/2f9657970276ff8cb10c1908968dbcde. As far as I can tell this path works. The problem is that no client certificate is sent during the discovery process.

With verify required, discovery fails with remote error: tls: certificate required in syncthing’s log (ie. one wasn’t sent (?)).

8

You can try to debug whats going on on the wire level using Wireshark, but if you have TLSv1.3 enabled you’re not going to see much. TLSv1.2 is much better for debugging.

If you have a look in the other thread, they used some other backend to debug whats being send there. You could try that too.

9

wireshark or tcpdump -A can show you the request towards the backend discovery server, as that part is not encrypted. A possible culprit could be that the discosrv doesn’t like the SSL certificate header for whatever reason.

1 Like
10

I didn’t have much success locating the client certificate request with wireshark. But I believe I can spot it with:

openssl s_client -connect stds.home:443

When verify is set to either optional or required in haproxy, the openssl command outputs Acceptable client certificate CA names and a list of DNs of CAs, which does not happen without the verify option.

From IBM Docs

If the server requires a digital certificate for client authentication, the server sends a “client certificate request” that includes a list of the types of certificates supported and the Distinguished Names of acceptable Certification Authorities (CAs).

Could you maybe point me to a relevant area in syncthing’s code. How is the client certificate request/offering handled?

11

It’s an http.Client with a TLS client certificate:

The rest is standard HTTPS/TLS connection handling in the Go standard library. We don’t do anything specific to send or not send the certificate apart from making it available to the HTTP client and then performing HTTP requests.

1 Like
12

FYI: https://github.com/haproxy/haproxy/issues/1399. Any input (in the github issue(?)) helpful. Does the GO side of the description seem valid?

13

I am like, 99.99% certain that it’s not a bug, but lack of knowledge/understanding on how to configure it properly.

TLS is a pretty well defined protocol, and given that whole world depends on it for security, I am very hesitant to believe that there is enough ambiguity to have “bad interaction with Go”.

We already have documentation for nginx/traefic etc, is switching to something where someone else already worked out how to do it not an option?

15

Yes I see your point (though 99.99% seems like a dangerously high certainty percentage :⁣) ). I reached a point where I did now know how to further proceed with the configuration, which is why I reported it. Of course it’s quite possible I might have missed something. Maybe the “bug” is simply one of missing or not visible enough documentation.

Yes, in any case I can knock myself out with nginx (I use haproxy for other stuff, hence the choice), or even not use a proxy. At this point I’m looking into this mostly out of persistence/interest, and just a little bit of a sense of public service! :⁣) Thanks for all the help and responses.

1 Like
16

Your Go reproducer looks sane to me.

17

Hi,

I’ve been running into pretty much exactly the same issue as described here and it got me wondering: is there any advantage transmitting devices’ certificates through the ssl protocol?

As far I can tell from the server code i read, the certificate is never used for the intended purpose (as far as SSL goes) which is validating the client’s identity against a CA and encrypting communication with it. It is only used as the basis to compute the device’s ID.

I have to admit it’s a pretty neat way of doing things. However it makes running a discovery server behind a reverse proxy much more complicated and requires custom configuration when at all possible. It is my understanding that sending the certificate through the request body would be equally secure and functional whilst allowing for a much easier setup. Is it a solution you would consider an issue/PR for?

18

Having it part of the handshake ensures that a client can only announce addresses for itself, not for any other random device id.

1 Like
19

Right, because the handshake process requires the private key as well. I somehow forgot about that.

Meaning that, whilst it is possible to transmit the certificate through the request body, it would also require a challenge system to prove ownership of the private key, making the whole thing a lot more complicated. Thanks for the answer.

1 Like
20

Hi – just curious – do you have an writeup or example on how to configure Traefik as a reverse proxy with the discovery server? I’ve got it working the an nginx reverse proxy (SWAG container), however since I’m working with docker, traefik would be an added bonus.

21