[SOLVED] Discovery server behind traefik (2.8)

Hi,

I’m trying to follow @cobertos & @kevdog and have Discovery Server behind traefik.

Summary:

  • POST made by Syncthing client seem to go through and populate Discovery Server.
  • GET made by Syncthing client do not seem to go through Traefik to Discovery Server as seem like are without client side cert (Syncthing client side)
  • GET made by CURL does go through Traefik to Discovery Server and response includes details as expected (json).

Therefore it is either misconfig at my end as it requires Client Cert or issue on Syncthing side that it doesn’t use cert for GET requests.

Is it maybe due to(Traefik yaml) - full config below:

tls:
  options:
    syncthing-discosvr:
      clientAuth:
        clientAuthType: RequireAnyClientCert

I don’t have it referenced as ‘@ file’ in traefik labels for docker-compose definitions though it seems like it is taken into account and maybe exactly what blocks the GET request without client cert? Or maybe it is rather on Syncthing side that it doesn’t use cert for GET request?

Details:

Current config follows:

By the looks what gets through Traefik to Discovery Server (tshark view):


Request:
Hypertext Transfer Protocol
    POST / HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): POST / HTTP/1.1\r\n]
            [POST / HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: POST
        Request URI: /
        Request Version: HTTP/1.1
    Host: st-discosvr..lets.encrypt\r\n
    User-Agent: Go-http-client/1.1\r\n
    Content-Length: 84\r\n
        [Content length: 84]
    Accept-Encoding: gzip\r\n
    Content-Type: application/json\r\n
    X-Client-Port: 443\r\n
    X-Forwarded-For: 192.168.110.95\r\n
    X-Forwarded-Host: st-discosvr..lets.encrypt\r\n
    X-Forwarded-Port: 443\r\n
    X-Forwarded-Proto: https\r\n
    X-Forwarded-Server: 3c8bd0e48a37\r\n
     [truncated]X-Forwarded-Tls-Client-Cert: MIICHTCC----cut----DBaMEo
    X-Real-Ip: 192.168.110.95\r\n
    \r\n
    [Full request URI: http://st-discosvr..lets.encrypt/]
    [HTTP request 1/1]
    File Data: 84 bytes
JavaScript Object Notation: application/json



Response:
Hypertext Transfer Protocol
    HTTP/1.1 204 No Content\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 204 No Content\r\n]
            [HTTP/1.1 204 No Content\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Response Version: HTTP/1.1
        Status Code: 204
        [Status Code Description: No Content]
        Response Phrase: No Content
    Reannounce-After: 3455\r\n
    Date: Wed, 28 Sep 2022 23:43:40 GMT\r\n
    \r\n
    [HTTP response 1/1]
    [Time since request: 0.000207722 seconds]

With that said, Traefik shows:

evel=debug msg="http: TLS handshake error from 192.168.110.95:33030: tls: client didn't provide a certificate"

And I wouldn’t be worried if the two clients using discovery server would see each other, but they don’t. Both are in different subnets but no nat and local discovery is turned of on purpose.

The moment I’d change Discovery Server from http (for traefik purposes) to normal https and use /?id= clients start to see each other. Tried adding /?id= when using trafeik but norhing changed.

These logs:

evel=debug msg="http: TLS handshake error from 192.168.110.95:33030: tls: client didn't provide a certificate"

Together with Discovery server seeing only type of entries (no GET)

78629a0f5f3f164f POST /

Suggest that maybe the POST works and is passed through, but when Syncthing client makes GET request it is blocked by Traefik (my misonfig probably requiring client cert.

Though when running

curl -X GET http://st-discosvr..lets.encrypt/?device=XXXX-CUT-XXXX --key ~/.config/syncthing/key.pem --cert ~/.config/syncthing/cert.pem -v
{"seen":"2022-09-29T00:13:23.921082645Z","addresses":["quic:// .....

Which got me confused completely, as it seems like for GET when cert supplied, Traefik passes it through, though Syncthing client doesn’t seem to send cert for GET requests? Could it be the case?

Regardless - hope is that @cobertos or @kevdog could advise what am I missing?

    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.syncthing-discosvr-rtr.rule=( (Host(`st-discosvr..lets.encrypt`)) || (Host(`st-discosvr.$DOMAIN`)) || (Host(`stds.$DOMAIN`)) )"
      - "traefik.http.routers.syncthing-discosvr-rtr.entrypoints=websecure,extwebsecure"
      - "traefik.http.routers.syncthing-discosvr-rtr.tls=true"
      - "traefik.http.routers.syncthing-discosvr-rtr.tls.options=syncthing-discosvr@file"
      ## HTTP Services
      - "traefik.http.routers.syncthing-discosvr-rtr.middlewares=syncthing-discosvr-passtlsclientcert,header-transform@file "
      - "traefik.http.middlewares.syncthing-discosvr-passtlsclientcert.passtlsclientcert.pem=true"
      - "traefik.http.routers.syncthing-discosvr-rtr.service=syncthing-discosvr-svc"
      - "traefik.http.services.syncthing-discosvr-svc.loadbalancer.server.port=8443"

http:
  middlewares:
    header-transform:
      plugin:
        header-transform-plugin:
          Rules:
            - Rule:
              Name: 'X-Client-Port Set'
              Header: 'X-Client-Port'
              Value: '^X-Forwarded-Port'
              HeaderPrefix: "^"
              Type: 'Set'

cat syncthing-discosvr-headers-transform.yaml syncthing-discosvr-tls.yaml 
http:
  middlewares:
    header-transform:
      plugin:
        header-transform-plugin:
          Rules:
            - Rule:
              Name: 'X-Client-Port Set'
              Header: 'X-Client-Port'
              Value: '^X-Forwarded-Port'
              HeaderPrefix: "^"
              Type: 'Set'
tls:
  options:
    syncthing-discosvr:
      clientAuth:
        clientAuthType: RequireAnyClientCert

Any help is highly appreciated!

Thank you.

It took plenty scratching my head and… reading my very own post to realize questions raised and coming up with switching from:

tls:
  options:
    syncthing-discosvr:
      clientAuth:
        clientAuthType: RequireClientCert

to:

tls:
  options:
    syncthing-discosvr:
      clientAuth:
        clientAuthType: RequireAnyClientCert

Looking again at @cobertos guide, I did spot my mistake… it has been tehre as RequireAnyClientCert, not sure where I did take RequireClientCert from.

Many thanks again for all documentation and earlier details under other posts!

Hey really glad you figured this out. It’s honestly been a long time since I’ve looked at this. Once I got it working, I kind of went on to other things and I haven’t really looked back. Took me awhile to review my notes and such on the topic.

Here is my dynamic configuration file. I’m honestly not sure if it makes any difference. I wanted however to use TLS1.3 to make it as secure as I possibly could.

http:

  middlewares:
    mw_compress_headers:
      compress: {}
    header-transform:
      plugin:
        header-transform-plugin:
          Rules:
            - Rule:
              Name: 'X-Client-Port Set'
              Header: 'X-Client-Port'
              Value: '^X-Forwarded-Port'
              HeaderPrefix: "^"
              Type: 'Set'

tls:
  options:
    default:
      minVersion: VersionTLS12
      sniStrict: true
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    intermediate: &intermediate_tls
      minVersion: VersionTLS12
      sniStrict: true
      cipherSuites:
        - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
        - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    modern: &modern_tls
      minVersion: VersionTLS13
      sniStrict: true
    syncthing-discosrv:
      <<: *modern_tls
      clientAuth:
        clientAuthType: RequireAnyClientCert

Here are my lables:

    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=net"
      ## HTTP Routers
      - "traefik.http.routers.syncthing-discosrv.rule=Host(`disco.<domain>.com`)"
      - "traefik.http.routers.syncthing-discosrv.tls=true"
      - "traefik.http.routers.syncthing-discosrv.tls.certresolver=le"
      - "traefik.http.routers.syncthing-discosrv.tls.domains[0].main=disco.<domain>.com"
      - "traefik.http.routers.syncthing-discosrv.tls.domains[0].sans=disco.<domain>.com"
      - "traefik.http.routers.syncthing-discosrv.entrypoints=web,websecure"
      - "traefik.http.routers.syncthing-discosrv.tls.options=syncthing-discosrv@file"
      - "traefik.http.routers.syncthing-discosrv.middlewares=syncthing-discosrv-middleware,header-transform@file"
      ## Middlewares
      - "traefik.http.middlewares.syncthing-discosrv-middleware.passtlsclientcert.pem=true"
      ## Services
      - "traefik.http.services.syncthing-discosrvr.loadbalancer.server.port=8443"

I’m not sure if that helps at all.

Thank you very much for updates! Yours and @cobertos saved me from re-inventing the wheel, or better… well, am not sure if I’d figure it out. It saved me for sure massive amount of time for research/tests as am not that expert in advanced Traefik settings.

Big thanks again!

@kevdog , I might have small question on which I can’t find answer easily (probably wrong search queries made from my end).

In your example, you’ve

    syncthing-discosvr:
      <<: *modern_tls

Which uses reference anchor used earlier:

    modern: &modern_tls

Question is how to use such reference across different yaml files? I like to keep app related settings within single files for easy portability and as above section defines default which I want to apply to other apps/traefik services, question is how to use that reference to modern_tls defined in different file?

Error occurred during watcher callback: ... yaml: unknown anchor 'modern_tls' referenced"

I have a feeling that it won’t work: https://yaml.org/spec/1.2-old/spec.html#id2765878

Is there any other way to keep definitions in one place and re-use in other files?

According to what I was reading on StackOverflow: Exporting anchors across YAML files - Stack Overflow

I don’t think there is a way for a yaml document to reference another yaml document. It doesn’t seem to be part of the yaml standard where all files are independent.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.