Syncthing Global discovery server behind traefik reverse proxy

kevdog · October 11, 2023, 3:58am

So for background I’m referencing this post which I’ve personally responded to running a global discovery server using the experimental header-transform plugin: [SOLVED] Discovery server behind traefik (2.8) - #5 by bugsyb

Fast forward to present day, and I’m getting an error within the disco server container:

stdiscosrv v1.23.4 "Fermium Flea" (go1.20.2 linux-amd64) teamcity@build.syncthing.net 2023-04-05 13:25:55 UTC [purego]
Server device ID is FRYCIDB-TKR3B57-2W52MDH-OPUG66Z-XIUTNO6-SXVQUKN-QJH52HQ-O5UEYQ5
4cb05d8ecca62cf2 POST /
4cb05d8ecca62cf2 no certificates: certificate decode result is empty

So to touch on a few topics – I’m building the disco server container from scratch via a docker file. I’m aware there is a version 1.5 out, however my docker file contains the command:

###############################
#           Build             #
###############################
ARG VERSION=v1.23.4
ENV DOWNLOAD_URL="https://github.com/syncthing/discosrv/releases/download/$VERSION/stdiscosrv-linux-amd64-$VERSION.tar.gz"
###############################

I’m not exactly sure where future versions of the discosrv tarballs are located since I can’t simply change the argument version to 1.50.0 or something like that. The syncthing github page isn’t helpful in locating this file or something else.

Anyway moving ahead: My traefik static configuration file is using the following rewrite headers plugin:

experimental:
  plugins:
    header-transform:
      moduleName: "github.com/adyanth/header-transform"
      version: "v1.0.0"

Within my traefik dynamic configuration file: I have the following middlewares and tls defined:

http:

  middlewares:
    header-transform-plugin:
      plugin:
        header-transform:
          Rules:
            - Rule:
              Name: 'X-Client-Port Set'
              Header: 'X-Client-Port'
              Value: '^X-Forwarded-Port'
              HeaderPrefix: "^"
              Type: 'Set'

And my actual docker file to create the container is:

  disco:
    build:
      context: .
      dockerfile: Dockerfile
    image: syncthing-discovery
    container_name: disco
    hostname: disco
    domainname: server.com
    restart: unless-stopped
    tty: True
    stdin_open: True
    networks:
      - net
      #npm-net:
      #  ipv4_address: 10.161.0.6
    ports:
      - 8443:8443
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=net"
      - "traefik.http.routers.syncthing-discosrv.rule=Host(`disco.server.com`)"
      - "traefik.http.routers.syncthing-discosrv.tls=true"
      - "traefik.http.routers.syncthing-discosrv.tls.certresolver=le"
      - "traefik.http.routers.syncthing-discosrv.tls.domains[0].main=disco.server.com"
      - "traefik.http.routers.syncthing-discosrv.tls.domains[0].sans=disco.server.com"
      - "traefik.http.routers.syncthing-discosrv.entrypoints=web,websecure"
      - "traefik.http.routers.syncthing-discosrv.tls.options=syncthing-discosrv@file"
      - "traefik.http.routers.syncthing-discosrv.middlewares=syncthing-discosrv-middleware,header-transform-plugin@file"
      - "traefik.http.middlewares.syncthing-discosrv-middleware.passtlsclientcert.pem=true"
      - "traefik.http.services.syncthing-discosrvr.loadbalancer.server.port=8443"
    environment:
      - TZ
      - PUID=6000
      - PGID=6000
    volumes:
      - /data/disco/db:/home/discosrv/db
      - /data/swag/etc/letsencrypt/live/disco.server.com/privkey.pem:/home/discosrv/certs/key.pem:ro
      - /data/swag/etc/letsencrypt/live/disco.server.com/fullchain.pem:/home/discosrv/certs/cert.pem:ro

So I’m not exactly sure how to examine the headers passed to the disco server container. I’m aware this was running once in the past however I’m kind of stumped now by the no certificates line.

bt90 · October 11, 2023, 5:27am

Sounds related:

github.com/syncthing/syncthing

traefik no longer url escape X-Forwarded-Tls-Client-Cert header

opened 08:08PM - 30 Sep 23 UTC

Aetf

bug

stdiscosrv has the ability to handle certificates passed by traefik proxy server… in `X-Forwarded-Tls-Client-Cert` header. Previously, traefik url encodes the content but it stopped doing so since version v2.9.4. This becomes troublesome for setups where stdiscosrv sits behind newer traefik versions, as now stdiscosrv tries to url [decode](https://github.com/syncthing/syncthing/blob/main/cmd/stdiscosrv/apisrv.go#L356) the header content when it is not encoded. The symptom can be very confusing, as if some client's certificate's pem text happens to not contain `+`, etc, then it works perfectly, but other clients will always fail announcement with 403 error. More details can be found at https://github.com/traefik/traefik/issues/9669. I'm not sure what should be the best strategy here to support both old and new traefik versions... but wanted to at least raise attention. stdiscosrv version: 1.24.0

kevdog · October 11, 2023, 2:02pm

@bt90 Really helpful link – not sure how you keep up on all the changes but great link. That’s just ducky – I just for now converted over to using SWAG container since it looks like this issue isn’t going to be resolved either on the traefik or disco server end. I guess this issue wouldn’t be resolved even if I actually passed a real client cert to the disco server container rather than kind of faking it as I’m doing now.

bt90 · October 11, 2023, 5:46pm

With the issue resolved you should be able to pass the real certificate via traefik to the discovery server.

kevdog · October 11, 2023, 7:41pm

wasn’t aware it’s resolved…assuming it is resolved…I would need to pass a client certificate to the container correct? I’m assuming the disco server would need to have the CA certificate that created the client certificate which because by default I’m using Le certificates with traefik would mean I could self signed client certificate signed through a self created CA authority. I’m assuming since the disco server is started in http mode it wouldn’t present any server certificates to the client for them to verify.

Right now I start my disco container with following options:

CMD ${USER_HOME}/server/discosrv \
    ${DISCO_OPTS} \
    -listen=":${SERV_PORT}" \
    -db-dir="${USER_HOME}/db/discosrv.db" \
    -cert="${USER_HOME}/certs/cert.pem" \
    -key="${USER_HOME}/certs/key.pem" \

Is there a CA option to add to the command line?

bt90 · October 12, 2023, 5:12am

It’s not resolved yet.

The requirements are pretty straight forward:

https://docs.syncthing.net/users/stdiscosrv.html#reverse-proxy-setup

kevdog · October 12, 2023, 12:30pm

Thanks for replying here. I’ve read the documentation you’ve listed here a lot. I guess the only part I’m misunderstanding is the client certificate section. I’m aware of the header declaration:

proxy_set_header X-SSL-Cert $ssl_client_cert;

From my more than rudimentary understanding of nginx, this references a variable @ssl_client_cert which I’m assuming is the PEM encoded client certificate – however nowhere in the syncthing setup (syncthing GUI client or reverse proxy) do I see an actual client certificate being defined. Is this passed by the syncthing client program in a hidden manner and not defined in the advanced options??

Lastly even more confused, I consulted the nginx doocumentation and found:

$ssl_client_cert returns the client certificate in the PEM format for an established SSL connection, with each line except the first prepended with the tab character; this is intended for the use in the proxy_set_header directive;

The variable is deprecated, the $ssl_client_escaped_cert variable should be used instead.

bt90 · October 12, 2023, 1:14pm

Syncthings protocol uses TLS encryption. Each node has its own certificate for this purpose. This is also used by the discovery server to validate that the client has a certificate that matches the ID it tries to register addresses for.

kevdog · October 12, 2023, 1:30pm

@bt90 - Ahhh Ok thanks. So the client certificate is “built-in” to each node. Ok makes sense. Thanks for explanation.

Hopefully the traefik url escape issue is resolved soon although I’m not holding my breath. Thanks for your time and knowledge.

AudriusButkevicius · October 12, 2023, 11:04pm

Its not built in, the device id is derived from cert.pem file (I think) in the config directory. Which is just the public certificate. The ownership of that certificate is proven by key.pem, which is the private key to that certificate. Anyone having that private key can claim that identity.

kevdog · October 13, 2023, 4:58am

Does a CA sign the CSR to create this cert.pem file? Who is the CA?

bt90 · October 13, 2023, 6:41am

It’s an entirely self-signed certificate. That’s why you need to add other devices by ID. Otherwise Syncthing can’t tell who to trust.

system · November 12, 2023, 6:42am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.