global@https://<domain>: Post https://<domain>: remote error: tls: illegal parameter

I’ve got an idea why that error might be happening: Because of certbot only being capable of getting rsa certs I’ve got an rsa cert and so my server can only support e.g. TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 while your server provides e.g. TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 and all of that paired with an ecc client cert (It stops after the frame the (among others) includes the client cert).

That’s just a thought and I can’t inspect more in wireshark, because of encryption and I don’t get the keylog (like in https://github.com/joneskoo/http2-keylog) implemented (I’m not using go, help appreciated).

I might try it with nginx, but I don’t know for sure as that involves downtime.

I guess maybe make sure your apache actually speaks and accepts ECDSA etc. Apart from that I have no idea. Our crypto settings aren’t especially cutting edge, the certs etc have been the same for many years. Our global nginx setup is many years old and using mostly defaults.

I guess maybe make sure your apache actually speaks and accepts ECDSA

It can’t because of certbot pending ecdsa support and you can’t use ecdas without an ecdsa cert (That has nothing to do with how new or old your nginx is).

If this is the problem this would be a problem with the golang client implementation.

There is no need for the server and client certs to be of the same type. If that were the case things would indeed break left and right.

That should be the case but that’s the only difference I can see in the ssl config right now

I can share you the address of the server if you want to test on your own

Sorry, no, I have no desire to troubleshoot this directly. I’m entirely satisfied with troubleshooting all the crap actually under my control :slight_smile:

The problem is as I do not know go I can’t even transform curl -v -X POST --cert cert.pem --key key.pem https://<redacted> -d '{"addresses":["tcp://0.0.0.0:22000"]}' -H "Content-Type: application/json" into go code to test if this already breaks

I’m able to reproduce this with the following code (copy-pasted together):

package main

import (
	"bytes"
	"crypto/tls"
	"fmt"
	"net/http"
	"os"
	"time"
)

func main() {
	kl, err := os.OpenFile("/tmp/keylog.txt", os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
	if err != nil {
		fmt.Printf("log: %s", err)
	}
	_, err = fmt.Fprintf(kl, "# SSL/TLS secrets log file, generated by go\n")
	if err != nil {
		fmt.Printf("log: %s", err)
	}
	fmt.Printf("Leaking TLS keys to /tmp/keylog.txt\n")

	cert, err := tls.LoadX509KeyPair(
		"/home/<redacted>/.config/syncthing/cert.pem",
		"/home/<redacted>/.config/syncthing/key.pem",
	)
	if err != nil {
		fmt.Printf("cert: %s", err)
	}

	var normalClient = &http.Client{
		Timeout: 5 * time.Minute,
		Transport: &http.Transport{
			Proxy: http.ProxyFromEnvironment,
			TLSClientConfig: &tls.Config{
				InsecureSkipVerify: true,
				KeyLogWriter:       kl,
			},
		},
	}

	var certClient = &http.Client{
		Timeout: 5 * time.Minute,
		Transport: &http.Transport{
			Proxy: http.ProxyFromEnvironment,
			TLSClientConfig: &tls.Config{
				InsecureSkipVerify: true,
				Certificates:       []tls.Certificate{cert},
				KeyLogWriter:       kl,
			},
		},
	}

	var jsonStr = []byte(`{"addresses":["tcp://0.0.0.0:22000"]}`)

	_, err = normalClient.Post("<redacted>", "application/json", bytes.NewBuffer(jsonStr))
	if err != nil {
		fmt.Printf("no-cert: %s", err)
	}

	_, err = certClient.Post("<redacted>", "application/json", bytes.NewBuffer(jsonStr))
	if err != nil {
		fmt.Printf("cert: %s", err)
	}
}

Log:

Leaking TLS keys to /tmp/keylog.txt
cert: Post https://<redacted>: remote error: tls: illegal parameter

So I’m going to report that to the golang tracker, because that seems a golang issue

Unless your theory is that no Go program can talk to an Apache server I would look closer at the Apache config.

There’s nothing to be seen, I reduced it to:

<VirtualHost *:443>
    # Skipped Name, logging and DocumentRoot

    Include includes/ssl.conf  # TLS certs
    SSLProtocol TLSv1.2
    SSLVerifyClient optional_no_ca
</VirtualHost>

It’s only happening with a client certificate and I’ve been able to reproduce this with only go std and it’s only happening with ecc certificates

Maybe your Apache doesn’t have support for ECC? There are all kinds of broken setups, FIPS compliant ssl libs, and so on. My point is that if we agree for the sake of argument that Go and Apache are generally compatible it boils down to configuration on either side. On the Go side there is not much set apart from the defaults. On the Apache side there are literally hundreds of variables. But feel free to file it as a Go bug.

A) curl works

B) See above there is ECDHE

C) It uses upstream openssl 1.1.1 and quite upstream apache (the ubuntu one, which essentially - if it has any changes - only has changes to the config file locations)

Filed https://github.com/golang/go/issues/33577

1 Like

Closed for reasons in the bug report