feat: startup and readiness probes for replicas

[leonardoce]

Extend the startup and readiness probes configured through the .spec.probes.startup and .spec.probes.readiness sections by adding two additional parameters:

• type: Defines the criteria for considering the probe successful. Accepted values include:
  • pg_isready: This setting marks the probe as successful when the pg_isready command exits with a status of 0. This is the default for both primary instances and replicas.
  • query: This setting marks the probe as successful when a basic query is executed locally on the postgres database.
  • streaming: This setting marks the probe successful when the replica starts streaming from its source and meets the specified lag requirements (details below).
• lag: Specifies the maximum acceptable replication lag, measured in bytes (expressed using Kubernetes quantities). This parameter is applicable only when type is set to streaming. If the lag parameter is not specified, the replica is considered successfully started/ready as soon as it begins streaming.

Consequently, the liveness probe has been streamlined to verify solely that the instance manager is operational, without monitoring the underlying PostgreSQL instance.

Closes: #6621

Release Notes

Improved Startup and Readiness Probes for Replicas: Enhanced support for Kubernetes startup and readiness probes in PostgreSQL instances, providing greater control over replicas based on the streaming lag.

[github-actions]

❗ By default, the pull request is configured to backport to all release branches.

• To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
• To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

[ardentperf]

IIUC, when replicas re-join they need to catch-up WAL - which has two effects:

1. clients can connect as soon as the readiness probe succeeds and during catch-up their queries may get results that lag more than usual.
2. the impetus for this PR: with preferred DataDurability (new feature in 1.25) all writes on the primary will completely hang during catch-up. the primary needs to write WAL synchronously and the latest WAL won't get acknowledged by the replica until it's caught up, leading to the hang on the primary.

+1 using a startup probe seems like a good idea

if someone is choosing preferred dataDurability then i don't think they would ever want the primary to hang. as a user, my hope is that replicas (re)joining a cluster should be as seamless as possible. a primary database hanging even briefly can be impactful on a large workload. I’m not sure we’d ever want a replica to be considered ready as soon as it starts streaming, when dataDurability is preferred. i suspect that users who choose preferred are specifically doing it because they want high availability.

my initial thought is that this should default to a small number of bytes and might not need to be configurable at all, unless as a setting for debugging or troubleshooting.

this would result in a slightly longer delay for replicas becoming ready even with dataDurability=required however it also means we eliminate the period on startup of higher-than-usual lag, which doesn't seem like a bad idea to me.

note: what i've written above is based on my understanding but i have not had time to test or verify it, so there might be mistakes. i haven't looked yet, but it could also be interesting to check how patroni approaches this.

[gbartolini]

IIUC, when replicas re-join they need to catch-up WAL - which has two effects:

1. clients can connect as soon as the readiness probe succeeds and during catch-up their queries may get results that lag more than usual.
2. the impetus for this PR: with preferred DataDurability (new feature in 1.25) all writes on the primary will completely hang during catch-up. the primary needs to write WAL synchronously and the latest WAL won't get acknowledged by the replica until it's caught up, leading to the hang on the primary.

That's correct.

[leonardoce]

e2e: https://github.com/EnterpriseDB/cloudnative-pg/actions/runs/12933848954

[ardentperf]

Is this PR in patroni solving a similar problem?

patroni/patroni#1786

[ardentperf]

interesting - from this recent pgcon talk it sounds like actually patroni might even still have the pause

https://youtu.be/CWrFPPG5USA?feature=shared&t=1190

i think in the talk he's focused on a 3rd node rejoining synchronous_standby_names with only 2 nodes required (so it doesn't cause the hang) - so i don't know for sure if there's any "special case" treatment for a synchronous cluster with only 2 nodes.

[leonardoce]

interesting - from this recent pgcon talk it sounds like actually patroni might even still have the pause [...]

Yes. I think there is no solution this problem but only compromises.
Having the cluster paused reduces the time needed for the replica to get in sync.
If the cluster is not paused the replica will need more time.
If may not even be able to get in sync again - depending on the workload of the primary.

[mnencia]

/test d=main tl=4

[github-actions]

@mnencia, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/13763633508