chore(liveness): fail liveness if db conns saturated for 10min

[iainsproat]

Description & motivation

Pods are placed in an unready state if the database connections are saturated for >20 seconds.

We currently have a situation where there are some long-running database queries. These cause the database connections available to a pod to be saturated for extended periods of time. This in turn causes the pod to be in an unready state. Over time, as the long-running queries accumulate, the number of pods which are unready increases. This eventually causes the entire cluster to be unavailable.

Currently the only way to recover is to restart the pod.

Including a check of saturated database connections in the liveness probe is a short-term hack to ensure affected pods are restarted whenever the encounter this issue.

Changes:

To-do before merge:

Screenshots:

Validation of changes:

Checklist:

• My pull request follows the guidelines in the Contributing guide?
• My pull request does not duplicate any other open Pull Requests for the same update/change?
• My commits are related to the pull request and do not amend unrelated code or documentation.
• My code follows a similar style to existing code.
• I have added appropriate tests.
• I have updated or added relevant documentation.

References

[fabis94]

🤷