I’m looking to use custom SSL certificates in my deployment. How does a Syncthing container verify certificates when forming an SSL connection?
I’m getting a * global@https://SyncthingDiscoveryServer:8385/: Post "https://SyncthingDiscoveryServer:8385/": tls: failed to verify certificate: x509: certificate signed by unknown authority error when trying to add it to my Syncthing client in a Docker container. I am able to add my discovery server to my PC without any issues. No certificate errors when accessing my Syncthing client in a Docker container from my PC from a browser and no issues adding these devices via their device IDs. I’m using non-default ports for the discovery server but mapping it to 8384.
My infrastructure looks like this:
My PC with root certs installed:
Syncthing Client
Docker Host:
Syncthing Client
Volume: /var/syncthing/config
cert.pem & https-cert.pem (both the exact same cert chain)
key.pem & https-key.pem (both the exact same cert chain)
Syncthing Discovery Server
/var/stdiscosrv
cert.pem
key.pem
I’ve deployed custom certificates from the same CAs for multiple applications already but for Syncthing, I’m scratching my head. I’m using the default paths for the certs and using the full cert chain (signed cert → intermediate CA → root).
I did some extra troubleshooting and on my PC’s Syncthing client added my Syncthing client in the Docker container as a discovery server just to see what would happen. Interestingly enough, I got this error that said * global@https://syncthing-docker:8384/: Post "https://syncthing-docker:8384/": tls: failed to verify certificate: x509: certificate is valid for syncthing-discovery-1, not syncthing-docker knowing full well that this wouldn’t work but it could provide useful troubleshooting information.
When two devices running Syncthing (not the discovery server) connect to each other, they check that the certificate of the remote device matches the device ID. The device ID is a representation of the certificate’s SHA256 hash, with check digits added. See Understanding Device IDs.
I’m not sure exactly how it verifies the certificates with discovery servers though.
Also why are you changing the cert.pem and key.pem? Syncthing as I explained earlier checks based on the SHA256 hash of the certificate, not based on who the CA is. This is often referred to as certificate pinning.
https-cert.pem and https-key.pem are for the web GUI and not the BEP/syncing part of Syncthing itself, and therefore it makes sense to use a properly signed certificate.
Also, using the same user certificates (i.e. certificates signed by the intermediate CA) for multiple things may confuse Syncthing.
Also, if you use the same user certificates on multiple devices, it will mess up Global Discovery and potentially Relaying, as there are multiple devices with the same device ID, and therefore Syncthing, the discovery server and the relay server see them as the same device.
You can get a random, unique sync protocol (BEP) certificate on each device by deleting the key.pem and cert.pem files and restarting Syncthing (do not do this on the discovery server). This will cause the device IDs to become unique, in case you made multiple devices have the same device ID by accident.
So for all 3 instances of Syncthing, they have their own set of cert.pem and key.pem files corresponding to the certificates I have issued for each of them. They are not sharing certificates.
I fixed the issue and what I ended up doing was destroying the Syncthing Discovery Server container, recreating it, and then recreating the certificate bundle for it . My suspicion is that I was moving too fast initially and put the wrong server certificate in the certificate bundle for the discovery server.
Certificate Bundles (cert.pem and https-cert.pem per instance):
Syncthing Instance or SyncthingDiscoveryServer Intermediate CA Root CA
You are right though; certificates should be unique between instances.
Note that only the cert.pem and key.pem are required to be different for every instance, as they are used to determine the device ID. Syncthing doesn’t use the https-cert.pem or https-key.pem to identify instances, so you could make them share the same wildcard certificate (and have the instances be subdomains of some master domain), for example.
Generally, it is still discouraged to reuse certificates or use wildcard certificates.
I may have just ran into a bug. With the discovery having a CA-signed cert and my Syncthing application in Docker having a self-signed cert, they are connecting to each other. I can’t make that happen with any other Syncthing instance with any sort of other configurations of certificates and deployment types.
I’ll open an issue on GitHub to debug shortly. At the very least, the question of how Syncthing trusts certificates should be answered.
I would love to see what updating the documentation looks like too if necessary.
CA-Signed Certificates, then back to Default/Generated Certificates
Syncthing-fork on Android (I understand that’s not supported here by Syncthing)
CA-Signed Certificates, then back to Default/Generated Certificates
cert.pem
certificate for device
Intermediate CA
Root CA
key.pem - corresponds to the certificate for the device in cert.pem
Syncthing in Docker 1 ↔ Syncthing Discovery Server
I got them to sync up with mismatched certficiate deployments. I tried to replicate it on Syncthing in Docker 2 with no success.
All other Syncthing deployments ↔ Syncthing Discovery Server
Original issue of * global@https://SyncthingDiscoveryServer:8385/: Post "https://SyncthingDiscoveryServer:8385/": tls: failed to verify certificate: x509: certificate signed by unknown authority
Syncthing in Docker 1 ↔ Syncthing installed on PC 1
Success, as expected with default/generated certificates
Syncthing in Docker 1 ↔ Syncthing installed on PC 2 (CA-Signed certificates)
Error appears in in the logs of Syncthing in Docker 1 saying Got peer certificate list of length 3 != 1 from peer at X.X.X.X:22000-Y.Y.Y.Y:22000/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-; protocol error
On the PC, I accidently pointed the address of the Remote device to Syncthing in Docker 1. I got an error saying “expected 1 certificate, got 3”. It would have never worked but gives some insight onto how certificates are handles.
Syncthing in Docker 1 ↔ Syncthing installed on PC 2 (Default/Generated certificates)
Success
Syncthing in Docker 1 ↔ Syncthing-fork on Android (CA-Signed certificates)
The Syncthing service wouldn’t start (yellow light) but I could login to the web GUI if I knew the password.
Because of the Syncthing service issue, I did not test syncing to the docker instance.
Syncthing in Docker 1 ↔ Syncthing-fork on Android
Success
The error with “Syncthing in Docker 1” to “Syncthing installed on PC 2” leads me to believe that Syncthing either does not handle a certificate bundle (multiple certs in cert.pem) properly.
My original issue was the verification error on the certificates. If only the first certificate is being considered in cert.pem and there is no other place to put place certificate authorities’ certificates for verification purposes, then it would make sense that these errors are appearing.
Syncthing-to-Syncthing connections use device IDs and never consider any signing chain, CA or not. There must be precisely one certificate presented by each side.
Syncthing-to-web-services like discovery servers will use regular HTTPS verification with CAs etc, unless you pass an id=... query option, which is to allow working with non-CA-signed discovery servers.
The two are different, you seem to perhaps be conflating them, and I’m not entirely sure what you’re trying to accomplish. If it’s just to connect to a custom discovery server that has a CA-signed certificate, that should work out of the box. If you need to add custom root CAs, you do that in the Docker image same as you would for curl or whatever else to work. That will never have any bearing, positive or negative, on the Syncthing-to-Syncthing connections.
I think I’m still confused here about the implementation of certificates with Syncthing. I understand that the https-cert.pem and https-key.pem files are used for the web GUI but I’m not sure if that plays any other role.
Assuming that the cert.pem and key.pem files are only used for Syncthing-to-Syncthing and Syncthing-to-web-servers (like discovery servers), I’m finding conflicting information.
The cert.pem and key.pem are the ones used for the sync protocol and should not be replaced.
All this to say and ask:
A certificate bundle as the https-cert.pem file seems to work properly for the web GUI and I can see my entire cert chain when I browse to my Syncthing instance.
What is the purpose of the cert.pem and the key.pem files for Syncthing and the Syncthing Discovery Servers?
I find the information you quote consistent, but there may be some sort of Stockholm syndrome happening.
You can do what you want with the GUI certificates for Syncthing (https-*.pem) and the discovery server’s certificate. Syncthing’s device/sync certificates must be single certs and not a certificate chain and, as mentioned, are always verified only by their hash (device ID).
I’m still not sure what it is you’re trying to accomplish, to be honest. Whatever it is you’re doing, I’d suggest you start from a working, out-of-the-box setup, and change one thing at a time while keeping it working. Changing many things and getting many errors is a good way to get confused about what’s going on.
Sorry if I wasn’t clear, but cert.pem and key.pem are the ones that are tied to the device ID.
Syncthing uses its key.pem file to prove that you have the correct key.pem for a given device ID (cert.pem), to the discovery server or a device you added for syncing.
The discovery servers use the key.pem and cert.pem files to prove authenticity to the client.
It may be possible to modify cert.pem and key.pem files to be properly signed, but it is not recommended or required, except on the discovery server. You can delete those and restart Syncthing to regenerate them, if you already changed them and want to start clean.
Note that this isn’t the case on the discovery server, those can and should be properly signed certificates or you may get errors. You can also change the path of the certificate and key files with the --cert and --key options on the discovery server, instead of using the default path.
. My suspicion is that I was moving too fast initially and put the wrong server certificate in the certificate bundle for the discovery server.