BUG: Poolers fail to start: Error failed to load certificate file "/etc/ssl/certs/pgbouncer.crt": Permission denied

[MarkCupitt]

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? Tactful Gerbal (latest)
• Where do you run it - cloud or metal? Kubernetes or OpenShift? Bare Metal K8s
• Are you running Postgres Operator in production? Yes
• Type of issue? Bug
  We have specified our own TLS Certificate, which works fine in DB Pods, however docs state if tls is specified in manifest, Pooler will load them and use them. We CAN see the cert mounted in the Pooler pods, however pods wont start with the following error, I understand the cert its trying to load is the self signed cert, which should be ignored, Im unable to check if it exists as pod never starts to get a shell

+ '[' pooler '=' postgres ]
+ '[' -z /tls/tls.crt ]
+ ln -s /tls/tls.crt /etc/ssl/certs/pgbouncer.crt
+ ln -s /tls/tls.key /etc/ssl/certs/pgbouncer.key
+ '[' '!' -z  ]
+ envsubst
+ envsubst
+ exec /bin/pgbouncer /etc/pgbouncer/pgbouncer.ini
2023-05-04 04:32:18.674 UTC [1] ERROR TLS setup failed: failed to load certificate file "/etc/ssl/certs/pgbouncer.crt": Permission denied
2023-05-04 04:32:18.674 UTC [1] FATAL TLS setup failed

Pooler pod spec Follows

apiVersion: v1
kind: Pod
metadata:
  name: bonanza-data-pooler-7c4796d8c9-j9rjl
  generateName: bonanza-data-pooler-7c4796d8c9-
  namespace: postgres-databases
  uid: 58cd609a-5e0f-4ff2-b7a5-bcbdbde590fc
  resourceVersion: '71452'
  creationTimestamp: '2023-05-04T04:32:16Z'
  labels:
    application: db-connection-pooler
    cluster-name: bonanza-data
    connection-pooler: bonanza-data-pooler
    pod-template-hash: 7c4796d8c9
    spilo-role: master
    team: bonanza
  annotations:
    cni.projectcalico.org/containerID: c8c2817e66e24d5bae8565f5aa45ce4d90adef3a9da28e05a485fb3963d25517
    cni.projectcalico.org/podIP: 10.1.110.20/32
    cni.projectcalico.org/podIPs: 10.1.110.20/32
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: bonanza-data-pooler-7c4796d8c9
      uid: cc3f6079-27f7-4ad0-92db-7ec740775f7d
      controller: true
      blockOwnerDeletion: true


spec:
  volumes:
    - name: tls
      secret:
        secretName: pg-tls
        defaultMode: 416
    - name: kube-api-access-44tp2
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: connection-pooler
      image: registry.opensource.zalan.do/acid/pgbouncer:master-27
      ports:
        - containerPort: 5432
          protocol: TCP
      env:
        - name: PGHOST
          value: bonanza-data
        - name: PGPORT
          value: '5432'
        - name: PGUSER
          valueFrom:
            secretKeyRef:
              name: postgres-pooler
              key: username
        - name: PGSCHEMA
          value: pooler
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-pooler
              key: password
        - name: CONNECTION_POOLER_PORT
          value: '5432'
        - name: CONNECTION_POOLER_MODE
          value: session
        - name: CONNECTION_POOLER_DEFAULT_SIZE
          value: '25'
        - name: CONNECTION_POOLER_MIN_SIZE
          value: '12'
        - name: CONNECTION_POOLER_RESERVE_SIZE
          value: '12'
        - name: CONNECTION_POOLER_MAX_CLIENT_CONN
          value: '10000'
        - name: CONNECTION_POOLER_MAX_DB_CONN
          value: '50'
        - name: CONNECTION_POOLER_CLIENT_TLS_CRT
          value: /tls/tls.crt
        - name: CONNECTION_POOLER_CLIENT_TLS_KEY
          value: /tls/tls.key
      resources:
        limits:
          cpu: 500m
          memory: 500Mi
        requests:
          cpu: 250m
          memory: 250Mi
      volumeMounts:
        - name: tls
          mountPath: /tls
        - name: kube-api-access-44tp2
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readinessProbe:
        tcpSocket:
          port: 5432
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        allowPrivilegeEscalation: false
  restartPolicy: Always
  terminationGracePeriodSeconds: 300
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  nodeName: e2-s1
  securityContext: {}
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

[MarkCupitt]

Related to #1230 by the looks

[MarkCupitt]

Directly related to #2201

[MarkCupitt]

Looks like it has been fixed here #2255

[MarkCupitt]

I also used these test versions

registry.opensource.zalan.do/acid/postgres-operator-test:1954846-pr-2255-5
registry.opensource.zalan.do/acid/pgbouncer:master-27

Can confirm problem is resolved ..