I’ve got a discovery server running in a Docker container on a VPS with port 8443 forwarded to the container. Nginx is also running on the VPS to reverse-proxy several subdomains to their respective containers (including Syncthing). I’m attempting to do that with the discovery container as well, but getting some errors.
Using https://domainname.com:8443/?<discosrv ID> as the discovery server works perfectly. (although how come the ‘discovery’ column in the GUI shows 3/3 with only the single server entry?)
Using https://discovery.domainname.com/?<discosrv ID> throws an error “tls: oversized record received with length 20527”. The server block shows 2/3 but discovery doesn’t seem to be working.
The nginx reverse proxy block is:
server {
listen 443;
server_name discovery.domainname.com;
location / {
include proxy_params;
proxy_pass https://localhost:8443;
}
}
syncthing@66c836829201:/discosrv$ ./discosrv -h
Usage of ./discosrv:
-cert="cert.pem": Certificate file
-db-backend="ql": Database backend to use
-db-dsn="memory://discosrv": Database DSN
-debug=false: Debug
-key="key.pem": Key file
-limit-avg=5: Allowed average package rate, per 10 s
-limit-burst=20: Allowed burst size, packets
-limit-cache=10240: Limiter cache entries
-listen=":8443": Listen address
-stats-file="": File to write periodic operation stats to
The discovery server needs a certificate and key file present in order to run. So is the SSL between syncthing (requesting access to the discovery server) and nginx, or between syncthing and the discovery server, or are those two separate SSL requests? Do I use two sets of keys/certs or the same set for nginx as the discovery server uses?
Sorry, one more question: does using the -http flag mean that I don’t need to use the id=<server ID> in the discovery server URL anymore since it’s proxied behind nginx SSL?
If your certificate is signed by a genuine CA and not self signed then you don’t.
Otherwise we provide the ID to do certificate pinning.
Be aware if you are using Nginx you will need to get Nginx to pass the base64 encoded client certificate in the X-SSL-Cert header, otherwise it won’t work, or you need to have the certificates on both Nginx and the discovery sever and not use -http option.
There is no docs for that yet, but googling around and checking nginx docs should explain how to do that.
You also need to pass the X-Forwarded-For header too, to pass clients IP to the discovery server.
Still unable to get this working. I have a CA-signed Let’s Encrypt certificate for the domain, I’m using the X-SSL-Cert header and the X-Forwarded-For header (complete settings below), and I’m running discosrv -http. It appears that GET requests are being properly received and delivering the expected ‘Not Found’ (404) in the discosrv debug log (since there are no entries). POST requests are being rejected with the ‘no certificates’ (403) error.
Nginx configuration for this domain:
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $http_x_forwarded_proto;
'' $scheme;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
default upgrade;
'' close;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"';
access_log off;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-SSL-Cert $ssl_client_cert;
server {
server_name <domain.name>;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
return 301 https://$host$request_uri;
}
server {
server_name <domain.name>;
listen 443 ssl http2 ;
access_log /var/log/nginx/access.log vhost;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:
DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:E
CDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA25
6:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA3
84:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS
-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA;
ssl_prefer_server_ciphers on;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
ssl_certificate /etc/nginx/certs/<domain.name>.crt;
ssl_certificate_key /etc/nginx/certs/<domain.name>.key;
ssl_dhparam /etc/nginx/certs/<domain.name>.dhparam.pem;
add_header Strict-Transport-Security "max-age=31536000";
include /etc/nginx/vhost.d/default;
location / {
proxy_pass http://<domain_name>;
}
}
Of you are testing with curl, it won’t work if you don’t provide client certificates. If you are using syncthing to test, something is wrong with the X-SSL-Cert header as discovery server is not getting the client certificate from nginx.
X-SSL-Cert is missing. I’m trying to figure out why that would be the case. Is that an nginx configuration problem? I’m stretching here…networking isn’t my strong point
I tried both $ssl_client_cert and $ssl_client_raw_cert in the nginx configuration.
Any other ideas? Thanks!
Scott
Edit: FWIW, I also checked the certificate using the tool at digicert.com and it didn’t show any certificate errors or warnings.
Edit2: I forgot to add that these headers are from syncthing client requests, not from curl. discosrv is running with the -http option.
Version 1.9.11 - I don’t see anything in the changelog for 1.9.12 that seems like a change. There’s very few useful references to X-SSL-Cert when googling ‘nginx “X-SSL-Cert”’
X-SSL-Cert is something we came up with.
Regardless, I don’t have access to discovery servers to check our config.
You need to play around with $ssl_client_cert and $ssl_client_raw_cert and get them presented to the discovery server. Sadly we don’t maintain Nginx to be able to help you with that, best you check on their forums/mailing lists/IRC channels.
So I switched tactics because it appears that somehow the X-SSL-Cert header is being completely discarded because of the contents of the cert. I can pass a different string as the header and it is passed through properly, but actually passing $ssl_client_cert or $ssl_client_raw_cert make the header completely disappear.
Now I’m trying without using the -http proxy flag and passing the same certs to discosrv that nginx uses.
(gdb) b querysrv.go:452
Breakpoint 1 at 0x40acd7: file /go/src/github.com/syncthing/discosrv/querysrv.go, line 452.
(gdb) run -key /etc/nginx/certs/sub.domain.com.key -cert /etc/nginx/certs/sub.domain.com.crt -debug
Still doesn’t work. I put a break in at line 452 in querysrv.go and when receiving a POST request from a syncthing client the certificates are still empty (I think):
Am I misunderstanding how to put this together? Nginx as https reverse proxy to discosrv with nginx using the same certificates in the server config as discosrv uses on startup, right?
Edit: I got some help from IRC#nginx to figure out that the $ssl_client_cert was being discarded…but we still don’t know why.
I mean, we are running it fine, so I am not sure where your problem lies.
If you run it without -http, the certificates are there. Perhaps you end up breaking on some other request, hence why the certificates are empty.
We use nginx to hand off encryption to something that’s written in C, so that there is less CPU load, and we read the PEM form client cert as handed to us by nginx.
What do you mean ‘…run with without -http, the certificates are there’? I ran with -http but with -key and -cert but there were still no certificates passed.
What other requests? Other requests to nginx? I also tested with only the discovery server running without any other of the containerized services so there shouldn’t have been any other requests going on.
Thanks for your patience…
Edit: in your nginx server block, do you set the ssl_verify_client or large_client_header_buffers?
Other requests meaning GET requests for example, which do not need to have a certificate.
As I said, I don’t have access to the discovery servers to check what our config looks like, but I think you might need to set something in nginx for it to tell the client that it’s supposed to provide the client certificate.
Well, that was painful, but it was the ssl_verify_client optional_no_ca; that finally did it. Thanks very much for the help! I learned a lot today…too bad I didn’t get anything else done
So, nginx (automatically configured with docker-gen and the letsencrypt-nginx-proxy-companion) reverse-proxying discosrv -http and both running in containers. Perfect!
Is the discovery server expecting a ‘client certificate’?