CA Certificates Out of Date and Not Working With LetsEncrypt

[isugimpy]

How are you running Renovate?

Self-hosted

Please select which platform you are using if self-hosting.

Gitea

If you're self-hosting Renovate, tell us what version of Renovate you run.

27.21.0

Describe the bug

Since the expiration of the CA certificates used by LetsEncrypt globally this morning, Renovate is failing to connect to my Gitea instance. I have confirmed that my cert is valid and that its full CA chain is as well. Looks like the container image is using an old version of the CA certificate bundle.

Relevant debug logs

Logs

ERROR: Repository has unknown error (repository=**redacted**)
       "err": {
         "task": {
           "commands": [
             "ls-remote",
             "--heads",
             "https://**redacted**@git.**redacted**
           ],
           "format": "utf-8"
         },
         "message": "fatal: unable to access 'https://git.**redacted**': server certificate verification failed. CAfile: none CRLfile: none\n",
         "stack": "Error: fatal: unable to access 'https://git.**redacted**': server certificate verification failed. CAfile: none CRLfile: none\n\n    at Object.action (/usr/src/app/node_modules/simple-git/src/lib/plugins/error-detection.plugin.ts:38:28)\n    at PluginStore.exec (/usr/src/app/node_modules/simple-git/src/lib/plugins/plugin-store.ts:24:29)\n    at /usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:114:40\n    at new Promise (<anonymous>)\n    at GitExecutorChain.handleTaskData (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:111:14)\n    at GitExecutorChain.<anonymous> (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:88:40)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.js:5:58)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)"
       }
DEBUG: Unknown res (repository=**redacted**)
       "res": "unknown-error"

Have you created a minimal reproduction repository?

No reproduction repository

[jokay]

Can confirm this as well, self-signed certificate works, Let's Encrypt wildcard certificate no more.

[isugimpy]

renovatebot/docker-renovate#235 has been opened for this as well. That's probably the more correct repo for it, but I'll leave this open because the visibility is important due to this being a critically breaking issue.

[viceice]

Can anybody check if a curl from that image works normally?

[viceice]

Essentially this is our base image: https://github.com/containerbase/buildpack

[rxbn]

Can anybody check if a curl from that image works normally?

I was able to curl my GitLab with Let's Encyrpt without a problem.

Edit: Using the current renovate docker image (renovate/renovate:latest)

[cschlesselmann]

Can anybody check if a curl from that image works normally?

curl from the renovate/renovate:latest image works fine

Logs

ubuntu@b560482c9209:/usr/src/app$ curl -v https://git.example.org
*   Trying 136.243.XXX.XXX:443...
* TCP_NODELAY set
* Connected to git.example.org (136.243.XXX.XXX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=git.example.org
*  start date: Aug  2 01:14:38 2021 GMT
*  expire date: Oct 31 01:14:36 2021 GMT
*  subjectAltName: host "git.example.org" matched cert's "git.example.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5617907c3550)
> GET / HTTP/2
> Host: git.example.org
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 302
< cache-control: no-cache
< content-type: text/html; charset=utf-8
< date: Thu, 30 Sep 2021 18:23:30 GMT
< location: https://git.example.org/users/sign_in
< referrer-policy: strict-origin-when-cross-origin
< server: Caddy
< server: nginx
< strict-transport-security: max-age=63072000
< x-content-type-options: nosniff
< x-download-options: noopen
< x-frame-options: DENY
< x-permitted-cross-domain-policies: none
< x-request-id: 01FGVZ4YEZF0JS2Z67KNGRT2NK
< x-runtime: 0.035325
< x-ua-compatible: IE=edge
< x-xss-protection: 1; mode=block
< content-length: 104
<
* Connection #0 to host git.example.org left intact
<html><body>You are being <a href="https://git.example.org/users/sign_in">redirected</a>.</body></html>

[isugimpy]

I can confirm that a curl works as well. Maybe node isn't using the standard ca-certificates package.

[isugimpy]

nodejs/node#4175 Possibly relevant.

[viceice]

No, it's not node, it the git checkout called by renovate.

So can anybody test git clone from our image?
Maybe it's only a git issue.

[isugimpy]

Confirmed. A git clone from inside the image is failing.

[jokay]

So this environment variable can be used as a temporary workaround?

GIT_SSL_NO_VERIFY="1"

[isugimpy]

Looks like that will work as a workaround, yep!

[rxbn]

So this environment variable can be used as a temporary workaround?

GIT_SSL_NO_VERIFY="1"

I can confirm that this workaround works! :)

[jokay]

Yes, but be aware it's not a safe workaround.

[viceice]

So we need to find out, why the git integrated curllib works different than the default Ubuntu curl. 🤔

[rarkins]

We're always on latest or near latest git version, and also on Ubuntu 20.04. It's very confusing.

Is anyone able to test if these workaround instructions for Ubuntu 14.04 are somehow relevant? https://jay.gooby.org/2021/09/30/remove-the-dst-root-ca-x3-crt-from-ubuntu-14-04-lts

[jokay]

I wonder why the gitlab-runner has no problems with the Git operations.

[rarkins]

I wonder why the gitlab-runner has no problems with the Git operations.

Renovate's gitlab runner? Or GitLab's?

[jokay]

Renovate's gitlab runner? Or GitLab's?

GitLab's runner. Well I use the Alpine Linux runner, not the Ubuntu one.

[rarkins]

Are there any publicly exposed git hosts we can test and verify against once we have ideas? e.g. one which will answer "invalid cert" now but hopefully "invalid credentials" once the cert problem is fixed?

[cschlesselmann]

https://try.gitea.io/ uses Letsencrypt as well

[philband]

We're always on latest or near latest git version, and also on Ubuntu 20.04. It's very confusing.

Is anyone able to test if these workaround instructions for Ubuntu 14.04 are somehow relevant? https://jay.gooby.org/2021/09/30/remove-the-dst-root-ca-x3-crt-from-ubuntu-14-04-lts

Can confirm removing the DST Root fixes it. Tested on the base image renovate build on (renovate/buildpack:5@sha256:3dc254c1cb49de3d7450bf6d7b883ad1effaa0cded2b5ec5ae326d9fd13da7fb)

Before

GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://git.example.org/testrepo.git
19:11:25.269641 git.c:455               trace: built-in: git clone https://git.example.org/testrepo.git
Cloning into 'testrepo'...
19:11:25.274169 run-command.c:666       trace: run_command: git remote-https origin https://git.example.org/testrepo.git
19:11:25.274933 git.c:743               trace: exec: git-remote-https origin https://git.example.org/testrepo.git
19:11:25.274967 run-command.c:666       trace: run_command: git-remote-https origin https://git.example.org/testrepo.git
19:11:25.278939 http.c:756              == Info: Couldn't find host git.example.org in the .netrc file; using defaults
19:11:35.378886 http.c:756              == Info:   Trying xx.xx.xx.xx:443...
19:11:35.378918 http.c:756              == Info: TCP_NODELAY set
19:11:35.411177 http.c:756              == Info: Connected to git.example.org (xx.xx.xx.xx) port 443 (#0)
19:11:35.428863 http.c:756              == Info: found 387 certificates in /etc/ssl/certs
19:11:35.428936 http.c:756              == Info: ALPN, offering h2
19:11:35.428961 http.c:756              == Info: ALPN, offering http/1.1
19:11:35.499862 http.c:756              == Info: SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
19:11:35.500233 http.c:756              == Info: server certificate verification failed. CAfile: none CRLfile: none
19:11:35.500267 http.c:756              == Info: Closing connection 0
fatal: unable to access 'https://git.example.org/testrepo.git/': server certificate verification failed. CAfile: none CRLfile: none

After

GIT_TRACE=1 git clone https://git.example.org/testrepo.git
19:14:19.837103 git.c:455               trace: built-in: git clone https://git.example.org/testrepo.git
Cloning into 'testrepo'...
19:14:19.841660 run-command.c:666       trace: run_command: git remote-https origin https://git.example.org/testrepo.git
19:14:19.842769 git.c:743               trace: exec: git-remote-https origin https://git.example.org/testrepo.git
19:14:19.842806 run-command.c:666       trace: run_command: git-remote-https origin https://git.example.org/testrepo.git
19:14:19.846685 http.c:756              == Info: Couldn't find host git.example.org in the .netrc file; using defaults
19:14:29.931415 http.c:756              == Info:   Trying xx.xx.xx.xx:443...
19:14:29.931451 http.c:756              == Info: TCP_NODELAY set
19:14:29.965915 http.c:756              == Info: Connected to git.example.org (xx.xx.xx.xx) port 443 (#0)
19:14:29.983242 http.c:756              == Info: found 384 certificates in /etc/ssl/certs
19:14:29.983320 http.c:756              == Info: ALPN, offering h2
19:14:29.983345 http.c:756              == Info: ALPN, offering http/1.1
19:14:30.080925 http.c:756              == Info: SSL connection using TLS1.3 / ECDHE_RSA_AES_256_GCM_SHA384
19:14:30.081446 http.c:756              == Info:         server certificate verification OK
19:14:30.081462 http.c:756              == Info:         server certificate status verification SKIPPED
19:14:30.081516 http.c:756              == Info:         common name: git.example.org (matched)
19:14:30.081550 http.c:756              == Info:         server certificate expiration date OK
19:14:30.081578 http.c:756              == Info:         server certificate activation date OK
19:14:30.081614 http.c:756              == Info:         certificate public key: EC/ECDSA
19:14:30.081641 http.c:756              == Info:         certificate version: #3
19:14:30.081654 http.c:756              == Info:         subject: CN=git.example.org
19:14:30.081682 http.c:756              == Info:         start date: Thu, 12 Aug 2021 21:32:31 GMT
19:14:30.081715 http.c:756              == Info:         expire date: Wed, 10 Nov 2021 21:32:29 GMT
19:14:30.081757 http.c:756              == Info:         issuer: C=US,O=Let's Encrypt,CN=R3
19:14:30.081788 http.c:756              == Info: ALPN, server accepted to use h2
19:14:30.081835 http.c:756              == Info: Using HTTP2, server supports multi-use
19:14:30.081866 http.c:756              == Info: Connection state changed (HTTP/2 confirmed)
19:14:30.081899 http.c:756              == Info: Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
19:14:30.082016 http.c:756              == Info: Using Stream ID: 1 (easy handle 0x55f46c6fcfd0)

[monrad]

It could be because of the openssl version used:

Unfortunately this does not apply to OpenSSL 1.0.2 which always prefers the untrusted chain and if that chain contains a path that leads to an expired trusted root certificate (DST Root CA X3), it will be selected for the certificate verification and the expiration will be reported.

From https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

[rarkins]

Seems like openssl also suggests surgically removing the old cert

[philband]

running apt update && apt upgrade -y pulls in a new version of ca-certificates ( 20210119~20.04.2), which also fixes the issue.
Changelog from ubuntu: http://changelogs.ubuntu.com/changelogs/pool/main/c/ca-certificates/ca-certificates_20210119~20.04.2/changelog

[viceice]

Ok, just on mobile yet. Will add the update to buildpack in around a hour.

[viceice]

The issue should be fixed with next base image update in next two or three hours.

Maybe you need to manually update to latest docker digest.

[viceice]

latest images are working again