Connection Pooler crashes when using custom TLS certificates

[heilerich]

Since #2146 has been merged connection pooler pods are going into crash loop on start. We were really looking forward to this feature, but sadly it isn't working at all in our environments. Pinging the authors of that PR @jeremie-seguin @FxKu

I suspect that I might have missed something here, because it looks to me like the current solution should have never worked anywhere. Anyways, in our deployments we are encountering the following:

The custom TLS certificates are mounted as a secret with a default mode of 0640. This creates two problems:

(1) The secret mount in the container is a separate tmpfs filesystem. The entrypoint.sh of the default pooler image (at tag master-26) tries to hardlink the files in the secret mount to the container's root overlay filesystem which produces a Cross-device link error. At that point the container will crash.

(2) Even with a fix for the above problem the container will probably still not run because it defaults to running as a non-root user. At the same time the operator does not set a fsGroup in the security context for the pods. This will cause the secret to be owned by root and with the operator setting the defaultMode of the secret to 0640 the certificates will not be readable by the pgbouncer process.

Possible solutions:

(1) Update the entrypoint.sh for the default pooler conatainer to use symlinks which will work across file system boundaries (i.e. replace ln with ln -s

(2) (a) Set a more permissive default mode for the secret mount, (b) use an initContainer running with UID 0 to copy and chown the certificates into a emptyDir volume (c) run pgboucner as UID 0 or (d) set a fsGroup for the pod.

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.9.0
• Where do you run it - cloud or metal? Kubernetes or OpenShift? KKP, Bare Metal K8s
• Are you running Postgres Operator in production? yes
• Type of issue? Bug report

[heilerich]

After fixing the above mentioned issues we additionally detected that the default paths for certificate and key files are assigned improper values here

postgres-operator/pkg/cluster/connection_pooler.go

Lines 351 to 356 in 30b6124

	if crtFile == "" {
	crtFile = "tls.crt"
	}
	if keyFile == "" {
	crtFile = "tls.key"
	}

This results in the key file being linked to the certificate destination and the key destination being linked to the containing folder instead of the key file.

[FxKu]

Sad to hear this new feature doesn't work as expected 😞 Could you provide a bugfix? My knowledge (and interest) in this whole certificates topic is very limited. This is our entrypoint atm:

#!/bin/sh

set -ex

if [ "$PGUSER" = "postgres" ]; then
    echo "WARNING: pgbouncer will connect with a superuser privileges!"
    echo "You need to fix this as soon as possible."
fi

if [ -z "${CONNECTION_POOLER_CLIENT_TLS_CRT}" ]; then
    openssl req -nodes -new -x509 -subj /CN=spilo.dummy.org \
        -keyout /etc/ssl/certs/pgbouncer.key \
        -out /etc/ssl/certs/pgbouncer.crt
else
    ln ${CONNECTION_POOLER_CLIENT_TLS_CRT} /etc/ssl/certs/pgbouncer.crt
    ln ${CONNECTION_POOLER_CLIENT_TLS_KEY} /etc/ssl/certs/pgbouncer.key
fi

envsubst < /etc/pgbouncer/pgbouncer.ini.tmpl > /etc/pgbouncer/pgbouncer.ini
envsubst < /etc/pgbouncer/auth_file.txt.tmpl > /etc/pgbouncer/auth_file.txt

exec /bin/pgbouncer /etc/pgbouncer/pgbouncer.ini

The symlink fix sounds like the quickest win for me. Why are the default values improper? Is it possible that these can be (and should stay) empty in some scenarios?

[heilerich]

I'll try to provide a fix for the operator if I can find some time in the next few days.

Why are the default values improper?

The crtFile = "tls.key" is set to the path of the key, which is just wrong. And the default value for keyFile is not set at all if the setting in the CR is the empty default.

What the fix will definitely require is changing the entrypoint to use symbolic instead of hard links.

15,16c15,16
<     ln ${CONNECTION_POOLER_CLIENT_TLS_CRT} /etc/ssl/certs/pgbouncer.crt
<     ln ${CONNECTION_POOLER_CLIENT_TLS_KEY} /etc/ssl/certs/pgbouncer.key
---
>     ln -s ${CONNECTION_POOLER_CLIENT_TLS_CRT} /etc/ssl/certs/pgbouncer.crt
>     ln -s ${CONNECTION_POOLER_CLIENT_TLS_KEY} /etc/ssl/certs/pgbouncer.key

[FxKu]

at least, the wrong default assignment I have fixed in #2216

[FxKu]

with #2255 merged and @heilerich verifying that it works with the changes I will close this PR. Thanks for raising it!