Custom global discovery server returning 403 Forbidden?

Hi, I’m trying to run a global discovery server for my devices at home using Docker and Traefik. I set it up on my server and PC in a configuration like the below:

PC ==(https)==> Traefik (on server) ==(http)==> Syncthing-Discosvr

Syncthingthing-Discosvr is using -http so that Traefik handles certs. On the PC though, I always seem to get 403 Forbidden as the error reported in the Syncthing GUI on the PC. It seems to be returned from Syncthing-Discosvr itself after looking at Traefik’s logs and trying curl -X POST https://st-ds.xxx.dev myself (which is what Syncthing itself seems to be doing).

Here’s all the configs/logs of interest.

Syncthing-Discosvr docker-compose snippet:

  syncthing-discosrv:
    image: syncthing/discosrv
    container_name: syncthing-discosrv
    networks:
      - traefik
    environment:
      PUID: 1000
      PGID: 1000
    volumes:
      - ./syncthing/DATA/syncthing-discosrv:/var/stdiscosrv
    restart: unless-stopped
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.syncthing-discosrv.loadbalancer.server.port=8443"
      - "traefik.http.routers.syncthing-discosrv.entrypoints=https"
      - "traefik.http.routers.syncthing-discosrv.rule=Host(`st-ds.xxx.dev`)"
      - "traefik.http.routers.syncthing-discosrv.tls.certresolver=cloudflarednsresolver"
    command:
      # Tell it to run in HTTP, otherwise it will complain that not HTTPS
      - "-http"
      - "-debug"

networks:
  traefik:
    external:
      name: infra_traefik-proxy

Syncthing-Discosvr logs:

syncthing-discosrv    | stdiscosrv v1.18.0 "Fermium Flea" (go1.16.5 linux-amd64) docker@syncthing.net 2021-06-21 20:53:50 UTC [noupgrade, purego]
syncthing-discosrv    | Server device ID is E75BUJW-QT4SSSM-47JKLJI-TR4IBIA-6G7V7T6-2VLPZ7H-ZLKIJFE-ZBAMFQG
syncthing-discosrv    | 78629a0f5f3f164f GET /?device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2
syncthing-discosrv    | 380704bb7b4d7c03 GET /?device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2
syncthing-discosrv    | 0866cb397916001e GET /?device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2
syncthing-discosrv    | 0c697f48392907a0 POST /
syncthing-discosrv    | 0c697f48392907a0 no certificates
syncthing-discosrv    | 68255aaf95e94627 GET /?device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2
syncthing-discosrv    | 28b621587cb3ad0b GET /?device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2

Traefik’s docker-compose config:

version: "3.6"
services:
  traefik:
    image: traefik:v2.5
    container_name: traefik
    restart: unless-stopped
    command:
      - "--log.level=DEBUG"

      # Turn on dashboard, configured to use Basic Auth in this contains labels
      # https://docs.traefik.io/operations/dashboard/#secure-mode
      - "--api.dashboard=true" # Defaults to port 8080

      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      # This needs to be the FULL NETWORK NAME. If not, Traefil will sometimes work
      # and sometimes give a bad gateway for containers (randomly every run)
      # https://stackoverflow.com/a/46439598/2759427
      - "--providers.docker.network=infra_traefik-proxy"

      - "--entrypoints.http.address=:80"
      - "--entrypoints.https.address=:443"
      - "--entrypoints.syncthing-tcp.address=:22000"
      - "--entrypoints.syncthing-udp.address=:22000/udp"

      - "--certificatesresolvers.cloudflarednsresolver.acme.email=xxx@xxx.com"
      - "--certificatesresolvers.cloudflarednsresolver.acme.storage=/data/acme.json"
      - "--certificatesresolvers.cloudflarednsresolver.acme.caServer=https://acme-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.cloudflarednsresolver.acme.dnschallenge.provider=cloudflare"

      - "--metrics.prometheus=true"
    volumes:
      - "/opt/batto-infra/traefik:/data"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    ports:
      - "80:80"
      - "443:443"
      # Syncthing
      - "22000:22000/tcp" # Syncing
      - "22000:22000/udp" # Syncing
    environment:
      CF_DNS_API_TOKEN: '${CLOUDFLARE_DDNS_XXX_DEV_API_KEY}'
    networks:
      - traefik-proxy
    labels:
      - "traefik.enable=true"
      #== Middlewares ==
      # LAN IP only
      - "traefik.http.middlewares.lan-only.ipwhitelist.sourcerange=127.0.0.1/32, 192.168.0.0/24, 192.168.1.0/24"
      # Gzip
      - "traefik.http.middlewares.http-compress.compress=true"
      # Basic auth
      - "traefik.http.middlewares.auth.basicauth.users=$TRAEFIK_HTPASSWD"
      # https redirect
      - "traefik.http.middlewares.https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.https-redirect.redirectscheme.permanent=true"

      # Global redirect all http to https
      - "traefik.http.routers.http-catchall.rule=HostRegexp(`{any:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=http"
      - "traefik.http.routers.http-catchall.middlewares=https-redirect"

      # == Traefik Dashboard config (traefik-ception) ==
      - "traefik.http.routers.api.rule=Host(`traefik.xxx.dev`) || Host(`traefik.localhost`)"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.middlewares=lan-only,auth"

networks:
  traefik-proxy:

Traefik logs sample (1 request from Syncthing to Syncthing-discosrv)

traefik    | time="2021-09-22T20:43:57Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"User-Agent\":[\"Go-http-client/1.1\"],\"X-Forwarded-Host\":[\"st-ds.xxx.dev:443\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"c448494d942d\"],\"X-Real-Ip\":[\"192.168.0.206\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"st-ds.xxx.dev:443\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.206:55740\",\"RequestURI\":\"/?device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2\",\"TLS\":null}"
traefik    | time="2021-09-22T20:43:57Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" ForwardURL="http://172.18.0.4:8443" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"User-Agent\":[\"Go-http-client/1.1\"],\"X-Forwarded-Host\":[\"st-ds.xxx.dev:443\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"c448494d942d\"],\"X-Real-Ip\":[\"192.168.0.206\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"st-ds.xxx.dev:443\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.206:55740\",\"RequestURI\":\"/?device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2\",\"TLS\":null}"
traefik    | time="2021-09-22T20:43:57Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept-Encoding\":[\"gzip\"],\"User-Agent\":[\"Go-http-client/1.1\"],\"X-Forwarded-Host\":[\"st-ds.xxx.dev:443\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"c448494d942d\"],\"X-Real-Ip\":[\"192.168.0.206\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"st-ds.xxx.dev:443\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.206:55740\",\"RequestURI\":\"/?device=Y3KJK7J-Z4CESDE-F74R563-GUHWCNT-WQOHSAT-PCEOYVM-FMBOGMT-4TV5TA2\",\"TLS\":null}"

I’m also investigating some discovery server issues. Try also sending a client certificate in the curl command.

$ curl -X POST <url> --key <key> --cert <cert>

For my testing, I used the ones generated by syncthing. In my case found in: ~/.config/syncthing/{key,cert}.pem.

Do the ‘no certificates’ erros go away?

1 Like

Nope, same error

cobertos@bepis:~/Seafile/projects/batto-snuggles-infra$ curl -X POST https://st-ds.xxx.dev --key ~/.config/syncthing/key.pem --cert ~/.config/syncthing/cert.pem 
Forbidden

Syncthing-discosvr logs:

syncthing-discosrv    | 4dba7b0f9da1d7eb POST /
syncthing-discosrv    | 4dba7b0f9da1d7eb no certificates

Traefik logs:

traefik    | time="2021-09-22T21:14:56Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.68.0\"],\"X-Forwarded-Host\":[\"st-ds.xxx.dev\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"c448494d942d\"],\"X-Real-Ip\":[\"192.168.0.206\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"st-ds.xxx.dev\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.206:56524\",\"RequestURI\":\"/\",\"TLS\":null}"
traefik    | time="2021-09-22T21:14:56Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.68.0\"],\"X-Forwarded-Host\":[\"st-ds.xxx.dev\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"c448494d942d\"],\"X-Real-Ip\":[\"192.168.0.206\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"st-ds.xxx.dev\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.206:56524\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://172.18.0.4:8443"
traefik    | time="2021-09-22T21:14:56Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.68.0\"],\"X-Forwarded-Host\":[\"st-ds.xxx.dev\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"c448494d942d\"],\"X-Real-Ip\":[\"192.168.0.206\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"st-ds.xxx.dev\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.206:56524\",\"RequestURI\":\"/\",\"TLS\":null}"

Does your proxy send the X-SSL-Cert to the backend (Syncthing Discovery Server — Syncthing v1 documentation)? You could analyze the proxy->backend HTTP Post with wireshark and inspect the headers.

Disclaimer: I’m new to syncthing, and I don’t know what Traefik does :⁣)

1 Like

As in the other thread; the server needs to request the certificate. In Nginx the incantation is ssl_verify_client optional_no_ca;. I don’t know it in Traefik.

2 Likes

Ah I see, I missed the steps regarding the headers.

It looks like Traefik has this data but in differently named headers and doesn’t support renaming headers. Though there’s also an experimental plugin for this I will have to try.

Ill report back if I get it to work

1 Like

Took me quite a while, but I was able to get Traefik to pass everything. I am still receiving no certificates though :confused:

After running:

curl -X POST https://st-ds.xxx.dev --key ~/.config/syncthing/key.pem --cert ~/.config/syncthing/cert.pem

Traefik logs:

traefik    | time="2021-09-23T02:52:27Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"POST\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/2.0\",\"ProtoMajor\":2,\"ProtoMinor\":0,\"Header\":{\"Accept\":[\"*/*\"],\"User-Agent\":[\"curl/7.68.0\"],\"X-Client-Port\":[\"443\"],\"X-Forwarded-For\":[\"192.168.0.206\"],\"X-Forwarded-Host\":[\"st-ds.xxx.dev\"],\"X-Forwarded-Port\":[\"443\"],\"X-Forwarded-Proto\":[\"https\"],\"X-Forwarded-Server\":[\"9a2b6d7cf939\"],\"X-Forwarded-Tls-Client-Cert\":[\"MIICHDCCAaKgAwIBAgIICSuKpnyYs6QwCgYIKoZIzj0EAwIwSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMB4XDTIxMDkyMjAwMDAwMFoXDTQxMDkxNzAwMDAwMFowSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEHqSm2CQncvrE9McNXk73gQD6p6IvoCqzGFM3n7BUo6mDg5ybbA846YdlhvGaP41o3prHK61j44Z5fuvFMHs7PjaVejETzfTXR6XR3Xjk9%2BdXabIhicMggZcXy5plCRRRo1UwUzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB%2FwQCMAAwFAYDVR0RBA0wC4IJc3luY3RoaW5nMAoGCCqGSM49BAMCA2gAMGUCMGlMcbYrmY69TRAzFwfQ6ckvKswFnYRYR0aVw867D16zMITrWxEhcfyY71iqOFaHggIxAOHO23Ps1ocpoHAu2SbYWxIqCVsSbO%2FsQoFAS%2FKtpKr56qvwQ0yOdbDwwy21998rmg%3D%3D\"],\"X-Real-Ip\":[\"192.168.0.206\"],\"X-Ssl-Cert\":[\"MIICHDCCAaKgAwIBAgIICSuKpnyYs6QwCgYIKoZIzj0EAwIwSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMB4XDTIxMDkyMjAwMDAwMFoXDTQxMDkxNzAwMDAwMFowSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEHqSm2CQncvrE9McNXk73gQD6p6IvoCqzGFM3n7BUo6mDg5ybbA846YdlhvGaP41o3prHK61j44Z5fuvFMHs7PjaVejETzfTXR6XR3Xjk9%2BdXabIhicMggZcXy5plCRRRo1UwUzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB%2FwQCMAAwFAYDVR0RBA0wC4IJc3luY3RoaW5nMAoGCCqGSM49BAMCA2gAMGUCMGlMcbYrmY69TRAzFwfQ6ckvKswFnYRYR0aVw867D16zMITrWxEhcfyY71iqOFaHggIxAOHO23Ps1ocpoHAu2SbYWxIqCVsSbO%2FsQoFAS%2FKtpKr56qvwQ0yOdbDwwy21998rmg%3D%3D\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"st-ds.xxx.dev\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"192.168.0.206:34596\",\"RequestURI\":\"/\",\"TLS\":null}"

which formatted is

{
  "Method": "POST",
  "URL": {
    "Scheme": "",
    "Opaque": "",
    "User": null,
    "Host": "",
    "Path": "/",
    "RawPath": "",
    "ForceQuery": false,
    "RawQuery": "",
    "Fragment": "",
    "RawFragment": ""
  },
  "Proto": "HTTP/2.0",
  "ProtoMajor": 2,
  "ProtoMinor": 0,
  "Header": {
    "Accept": ["*/*"],
    "User-Agent": ["curl/7.68.0"],
    "X-Client-Port": ["443"],
    "X-Forwarded-For": ["192.168.0.206"],
    "X-Forwarded-Host": ["st-ds.xxx.dev"],
    "X-Forwarded-Port": ["443"],
    "X-Forwarded-Proto": ["https"],
    "X-Forwarded-Server": ["9a2b6d7cf939"],
    "X-Forwarded-Tls-Client-Cert": ["MIICHDCCAaKgAwIBAgIICSuKpnyYs6QwCgYIKoZIzj0EAwIwSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMB4XDTIxMDkyMjAwMDAwMFoXDTQxMDkxNzAwMDAwMFowSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEHqSm2CQncvrE9McNXk73gQD6p6IvoCqzGFM3n7BUo6mDg5ybbA846YdlhvGaP41o3prHK61j44Z5fuvFMHs7PjaVejETzfTXR6XR3Xjk9+dXabIhicMggZcXy5plCRRRo1UwUzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwFAYDVR0RBA0wC4IJc3luY3RoaW5nMAoGCCqGSM49BAMCA2gAMGUCMGlMcbYrmY69TRAzFwfQ6ckvKswFnYRYR0aVw867D16zMITrWxEhcfyY71iqOFaHggIxAOHO23Ps1ocpoHAu2SbYWxIqCVsSbO/sQoFAS/KtpKr56qvwQ0yOdbDwwy21998rmg=="],
    "X-Real-Ip": ["192.168.0.206"],
    "X-Ssl-Cert": ["MIICHDCCAaKgAwIBAgIICSuKpnyYs6QwCgYIKoZIzj0EAwIwSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMB4XDTIxMDkyMjAwMDAwMFoXDTQxMDkxNzAwMDAwMFowSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEHqSm2CQncvrE9McNXk73gQD6p6IvoCqzGFM3n7BUo6mDg5ybbA846YdlhvGaP41o3prHK61j44Z5fuvFMHs7PjaVejETzfTXR6XR3Xjk9+dXabIhicMggZcXy5plCRRRo1UwUzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwFAYDVR0RBA0wC4IJc3luY3RoaW5nMAoGCCqGSM49BAMCA2gAMGUCMGlMcbYrmY69TRAzFwfQ6ckvKswFnYRYR0aVw867D16zMITrWxEhcfyY71iqOFaHggIxAOHO23Ps1ocpoHAu2SbYWxIqCVsSbO/sQoFAS/KtpKr56qvwQ0yOdbDwwy21998rmg=="]
  },
  "ContentLength": 0,
  "TransferEncoding": null,
  "Host": "st-ds.xxx.dev",
  "Form": null,
  "PostForm": null,
  "MultipartForm": null,
  "Trailer": null,
  "RemoteAddr": "192.168.0.206:34596",
  "RequestURI": "/",
  "TLS": null
}

EDIT:

Output when syncthing/discosrv is replaced with ealen/echo-server (so this is exactly what the container sees)

{
  "name": "echo-server",
  "hostname": "2ebbcd088c6b",
  "pid": 1,
  "level": 30,
  "host": {
    "hostname": "st-ds.xxx.dev",
    "ip": "::ffff:172.18.0.2",
    "ips": []
  },
  "http": {
    "method": "POST",
    "baseUrl": "",
    "originalUrl": "/",
    "protocol": "http"
  },
  "request": {
    "params": {},
    "query": {},
    "cookies": {},
    "body": {},
    "headers": {
      "host": "st-ds.xxx.dev",
      "user-agent": "curl/7.68.0",
      "content-length": "0",
      "accept": "*/*",
      "x-client-port": "443",
      "x-forwarded-for": "192.168.0.206",
      "x-forwarded-host": "st-ds.xxx.dev",
      "x-forwarded-port": "443",
      "x-forwarded-proto": "https",
      "x-forwarded-server": "9a2b6d7cf939",
      "x-forwarded-tls-client-cert": "MIICHDCCAaKgAwIBAgIICSuKpnyYs6QwCgYIKoZIzj0EAwIwSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMB4XDTIxMDkyMjAwMDAwMFoXDTQxMDkxNzAwMDAwMFowSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEHqSm2CQncvrE9McNXk73gQD6p6IvoCqzGFM3n7BUo6mDg5ybbA846YdlhvGaP41o3prHK61j44Z5fuvFMHs7PjaVejETzfTXR6XR3Xjk9%2BdXabIhicMggZcXy5plCRRRo1UwUzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB%2FwQCMAAwFAYDVR0RBA0wC4IJc3luY3RoaW5nMAoGCCqGSM49BAMCA2gAMGUCMGlMcbYrmY69TRAzFwfQ6ckvKswFnYRYR0aVw867D16zMITrWxEhcfyY71iqOFaHggIxAOHO23Ps1ocpoHAu2SbYWxIqCVsSbO%2FsQoFAS%2FKtpKr56qvwQ0yOdbDwwy21998rmg%3D%3D",
      "x-real-ip": "192.168.0.206",
      "x-ssl-cert": "MIICHDCCAaKgAwIBAgIICSuKpnyYs6QwCgYIKoZIzj0EAwIwSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMB4XDTIxMDkyMjAwMDAwMFoXDTQxMDkxNzAwMDAwMFowSjESMBAGA1UEChMJU3luY3RoaW5nMSAwHgYDVQQLExdBdXRvbWF0aWNhbGx5IEdlbmVyYXRlZDESMBAGA1UEAxMJc3luY3RoaW5nMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEHqSm2CQncvrE9McNXk73gQD6p6IvoCqzGFM3n7BUo6mDg5ybbA846YdlhvGaP41o3prHK61j44Z5fuvFMHs7PjaVejETzfTXR6XR3Xjk9%2BdXabIhicMggZcXy5plCRRRo1UwUzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB%2FwQCMAAwFAYDVR0RBA0wC4IJc3luY3RoaW5nMAoGCCqGSM49BAMCA2gAMGUCMGlMcbYrmY69TRAzFwfQ6ckvKswFnYRYR0aVw867D16zMITrWxEhcfyY71iqOFaHggIxAOHO23Ps1ocpoHAu2SbYWxIqCVsSbO%2FsQoFAS%2FKtpKr56qvwQ0yOdbDwwy21998rmg%3D%3D",
      "accept-encoding": "gzip"
    }
  },
  "environment": {
    "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "HOSTNAME": "2ebbcd088c6b",
    "PORT": "8443",
    "PUID": "1000",
    "PGID": "1000",
    "NODE_VERSION": "14.17.1",
    "YARN_VERSION": "1.22.5",
    "HOME": "/root"
  },
  "msg": "Thu, 23 Sep 2021 03:42:12 GMT | [POST] - http://st-ds.xxx.dev/",
  "time": "2021-09-23T03:42:12.840Z",
  "v": 0
}

X-Forwarded-For, X-Client-Port, and X-Ssl-Cert are set and look correct.

Any ideas? Maybe Traefik encodes the certificate different than expected, PassTLSClientCert - Traefik ?

You’re confusing the discovery server because X-SSL-Cert is the Nginx header but the contents are not in the format Nginx uses. However you’re in luck because the discovery server also understands Traefik’s X-Forwarded-TLS-Client-Cert. So remove the X-SSL-Cert header, I guess.

Please file an update to Syncthing Discovery Server — Syncthing v1 documentation with a working config when you’ve sorted this out, to pay it forward. :slight_smile:

3 Likes

Ooohh, nice! I removed the X-Ssl-Cert header and was able to get past at least no certificates

syncthing-discosrv    | 60d469ad5e42a3e1 POST /
syncthing-discosrv    | 60d469ad5e42a3e1 decode: EOF

Oh I spoke too soon, it seems like that’s because I wasn’t sending any JSON in my curl

After hooking up a real Syncthing client to the global discovery server it works!

1 Like