deployment: add startupProbe for nfd-master #1810
Conversation
This patch mitigates inadvertent termination of nfd-master pods by the liveness probe on big clusters. With a recent change nfd-master started to wait (block) for informer caches to sync before starting the main loop. Consequently, this change also made the gRPC health enpoint to not respond until the caches have been synced. In big clusters the syncing the NodeFeature object cache takes a long time as the objects are big and there's (at least) one per each node in the cluster. Thus, in big clusters, the liveness probe kicks in and kills the nfd-master pod before it's ready.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: marquiz The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
cncf-cla: yes
✅ Deploy Preview for kubernetes-sigs-nfd ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
/assign @ArangoGutierrez |
|
@marquiz: GitHub didn't allow me to request PR reviews from the following users: lxlxok. Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
| @@ -144,8 +144,9 @@ API's you need to install the prometheus operator in your cluster. | |||
| | `master.config` | dict | | NFD master [configuration](../reference/master-configuration-reference) | | |||
| | `master.args` | array | [] | Additional [command line arguments](../reference/master-commandline-reference.md) to pass to nfd-master | | |||
| | `master.revisionHistoryLimit` | integer | | Specify how many old ReplicaSets for this Deployment you want to retain. [revisionHistoryLimit](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#revision-history-limit) | | |||
| | `master.livenessProbe` | dict | {"grpc":{"port":8082},"initialDelaySeconds":10} | NFD master pod [liveness probe](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/#liveness-probe) | | |||
| | `master.readinessProbe` | dict | {"grpc":{"port":8082},"initialDelaySeconds":5,"failureThreshold": 10} | NFD master pod [readiness probe](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/#readiness-probe)| | |||
| | `master.startupProbe` | dict | {"grpc":{"port":8082},"failureThreshold": 30} | NFD master pod [startup probe](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/#readiness-probe) | | |||
There was a problem hiding this comment.
link is wrong
| | `master.startupProbe` | dict | {"grpc":{"port":8082},"failureThreshold": 30} | NFD master pod [startup probe](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/#readiness-probe) | | |
| | `master.startupProbe` | dict | {"grpc":{"port":8082},"failureThreshold": 30} | NFD master pod [startup probe](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/#startup-probe) | |
|
ping @ArangoGutierrez |
k8s-ci-robot
added
the
needs-rebase
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This patch mitigates inadvertent termination of nfd-master pods by the liveness probe on big clusters.
With a recent change nfd-master started to wait (block) for informer caches to sync before starting the main loop. Consequently, this change also made the gRPC health enpoint to not respond until the caches have been synced. In big clusters the syncing the NodeFeature object cache takes a long time as the objects are big and there's (at least) one per each node in the cluster. Thus, in big clusters, the liveness probe kicks in and kills the nfd-master pod before it's ready.