APT error while updating

Fehl:8 https://apt.syncthing.net syncthing Release
Certificate verification failed: The certificate is NOT trusted. The certificate chain uses expired certificate. Could not handshake: Fehler bei der Verifizierung des Zertifikats. [IP: 2a03:b0c0:0:1010::99:8001 443]

However, I’ve installed the latest key etc. per https://apt.syncthing.net

The error is persitent since yesterday 19:00 h CEST - several futile attempts. Txs mgw

Your OS is probably a bit dated.

I’m on Ubuntu 20.04 LTS latest release - I’ll hesitate to call that outdated…

The certificate uses the correct root CA(ISRG Root X1) and is valid.

Could you run the following check?

openssl s_client -connect apt.syncthing.net:443

My Ubuntu 20.04 install connects without any problems.

I believe apt (at least on Debian) uses GnuTLS not OpenSSL, and GnuTLS is sadly horribly broken (also this, plus a dozen more links to issues). I believe the GnuTLS version shipped by Ubuntu is simply unable to validate Let’s Encrypts new “expired” certificate chain.

We might need to ask @calmh to configure the server to serve Let’s Encrypts “alternate chain” (which doesn’t include the cross-sign up the expired DST Root CA X3), which will fix connections with broken OpenSSL/GnuTLS.

The cross-sign to the expired root is only useful for Android devices, and I have doubts that there are many users connecting to apt via Android.

Edit: I completly forgot that Ubuntu backported patches to GnuTLS too, so a fully up-to-date 20.04 should be able to connect to apt.s.n

I tried our installation instructions in Docker images of Ubuntu 14.04, 16.04, 18.04 and 20.04. In 14.04 apt-get fails with a handshake error, in 16.04 and newer it works just fine out of the box. Perhaps the Docker images are not representative somehow?

% docker run -it --rm ubuntu:16.04 bash -c "apt-get update &&
  apt-get install -y curl apt-transport-https &&
  curl -s -o /usr/share/keyrings/syncthing-archive-keyring.gpg https://syncthing.net/release-key.gpg &&
  echo \"deb [signed-by=/usr/share/keyrings/syncthing-archive-keyring.gpg] https://apt.syncthing.net/ syncthing stable\" | tee /etc/apt/sources.list.d/syncthing.list &&
  apt-get update &&
  apt-get install syncthing &&
  syncthing --version"
syncthing v1.18.2 "Fermium Flea" (go1.17 linux-amd64) deb@build.syncthing.net 2021-08-22 18:04:47 UTC [noupgrade]
1 Like

Yeah that are the backports I forgot about. Last month they shipped patches to all supported versions which mitigate this issue. It’s a bit embarrassing that I completly forgot the backports, because I actually talked to the devs making these backports… Let’s just blame it on not enough sleep.

It still holds that the cross-sign is only useful for Android devices, so the apt.s.n endpoint is probably better off without it - it will help all machines not getting these fixes.

1 Like

apt.s.n runs a (very old) Caddy 1.something, which I tried upgrading to a recent Caddy yesterday but didn’t get all the nuances correctly ported to the new config format, and then I dropped it when I realized the problem wasn’t that Caddy was producing a too-old certificate chain. If there’s an incantation in new or old Caddy that helps then we could add that incantation, but otherwise it seems to me like an up to date system of even the old distributions should work fine with the current setup…

I’m also sorely tempted to outsource this problem and just throw the whole apt.s.n archive up into Azure/Netlify/GitHub pages or whatever and let them serve it… It’s just static files after all, with a few hundred gigs per month outgoing traffic.

1 Like

caddy2 also has support for ZeroSSL which might not be affected.

Edit: Possible workaround for those who are currently affected by this:

Comment out Syncthings repo in your apt sources, install all updates and reenable it again.

1 Like

It’s also possible to speak HTTP to apt.s.n, for legacy apt reasons.


Yeah ZeroSSL is a reseller for Sectigo and their roots still have some years left of lifetime (+ are also very old, so highly compatible). I do however consider switching the CA a “in emergency break glass” solution, and I don’t see the emergency right now (and I say this even though I’ve answered over 250 help topics about “all my systems broke just now, it says certificate expired” in the last ~2 days). ZeroSSL also has some questionable business practices. In addition, calmh said this is running Caddy v1, so it’s not possible anyway.

1 Like

Here you go:

~ $ openssl s_client -connect apt.syncthing.net:443
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = apt.syncthing.net
verify return:1
Certificate chain
 0 s:CN = apt.syncthing.net
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
Server certificate
subject=CN = apt.syncthing.net

issuer=C = US, O = Let's Encrypt, CN = R3

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
SSL handshake has read 4173 bytes and written 373 bytes
Verification: OK
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
Post-Handshake New Session Ticket arrived:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 002184D2610707448C6672A056F859E86B28C77AB3271607D44A4F98396FE9FE
    Resumption PSK: CD0088D587C40F45512BF490353F3027C6154CCB09BCBB0DF6A2673ABC12E00A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - ab f4 a5 48 cf a5 05 78-31 d3 9c b6 50 8d d8 0e   ...H...x1...P...
    0010 - ad 17 8e c5 93 4a 0d a1-e7 f3 e7 c9 57 0f 19 b1   .....J......W...
    0020 - 7a 13 db fd a4 90 d3 b4-80 9a 61 46 4f 2d 6d 38   z.........aFO-m8
    0030 - 8d b3 86 22 b1 98 b3 16-70 9f 6e b5 b4 ec c6 42   ..."....p.n....B
    0040 - d1 a0 34 70 31 8a 6c bb-a2 70 f4 90 10 f0 c5 03   ..4p1.l..p......
    0050 - 31 e1 10 90 3e 7c b2 a7-e4 dd c0 a0 32 41 dc d9   1...>|......2A..
    0060 - 0e e7 cd 09 04 e4 aa 15-33 5d f9 52 69 73 ca f9   ........3].Ris..
    0070 - 3b                                                ;

    Start Time: 1633192733
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
read R BLOCK

1 Like

Txs much. Switching to http solved the issue:

Holen:14 http://apt.syncthing.net syncthing InRelease [15,1 kB]
Holen:17 http://apt.syncthing.net syncthing/stable i386 Packages [2.224 B]
Holen:18 http://apt.syncthing.net syncthing/stable amd64 Packages [2.223 B]

It appears that after the latest updates this morning the error is gone for Ubuntu (several flavors) 20.04 kernel in use now 5.4.0-88-generic x86_64.

on Ubuntu 18.04 lts:

  1. apt update (with syncthing errors)
  2. apt install ca-certificates
  3. apt update (no errors)
  4. apt upgrade
1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.