Sidecar "replication-cert-copy" floods logging stack when autogrow annotation is triggered

[SebastianHuckeAdesso]

Please ensure you do the following when reporting a bug:

• Provide a concise description of what the bug is.
• Provide information about your environment.
• Provide clear steps to reproduce the bug.
• Attach applicable logs. Please do not attach screenshots showing logs unless you are unable to copy and paste the log data.
• Ensure any code / output examples are properly formatted for legibility.

Note that some logs needed to troubleshoot may be found in the /pgdata/<CLUSTERNAME>/pg_log directory on your Postgres instance.

An incomplete bug report can lead to delays in resolving the issue or the closing of a ticket, so please be as detailed as possible.

If you are looking for general support, please view the support page for where you can ask questions.

Thanks for reporting the issue, we're looking forward to helping you!

Overview

When the pgdata folder exceeds 75% disk usage, the sidecar replication-cert-copy triggers an autogrow annotation on the pod via curl:

postgres-operator/internal/postgres/config.go

Line 359 in 11210f8

curl --cacert "${CACERT}" --header "Authorization: Bearer ${TOKEN}" -XPATCH "${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods/${HOSTNAME}?fieldManager=kubectl-annotate" -H "Content-Type: application/json-patch+json" --data "${d}"

This is done every 5 seconds until the pgdata directory is expanded (either via autogrow or manually). When you collect the logs (via cluster log forwarder) and store them in an Elasticsearch database, this amounts to 2 GiB/day of log storage for an installation with two replicas.

Environment

Please provide the following details:

• Platform: OpenShift
• Platform Version: (Kubernetes: V1.30.14; OpenShift: 4.14.11)
• PGO Image Tag: (Version 5.8.3; registry.connect.redhat.com/crunchydata/postgres-operator@sha256:682ff2e1229c5afb1bf4bee7019353bdaad55a723218361a0e989443d3e7fbc3)
• Postgres Version (17.6)
• Storage: (Storageclass: esxopenshift-197-csi)

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

1. Let your pgdata folder grow until 75% of disk space are used. (Or set the initial pvc size small enough.)
2. Check the log output of the sidecar replication-cert-copy.

EXPECTED

1. Autogrow annotation is added once.
2. The request's response body is not written to stdout (unless in debug mode) -> either by outputting to /dev/null or by disabling autogrow annotation when it is already set.

ACTUAL

1. Every 5 seconds, the sidecar calls the api-server (see above) to annotate the pod again
2. It dumps the curl output to stdout. (i.e. json'ified pod yaml)
3. Result in Elasticsearch: separate log entries for each line that was written to stdout (even for a simple }) with additional K8s information (annotations, labels, networks etc.)

Logs

Original/plain container logs are not available, since we already expanded the PVC.

Log entries in Elasticsearch/Kibana:

Additional Information

None.