You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] Controller emits zero Warning events on NodeReadinessRule or Node objects when taint operations fail #362
The controller emits Kubernetes Events for successful taint operations on Node objects, but emits no Warning events when reconciliation or taint operations fail. Additionally, no events are ever emitted on the owning NodeReadinessRule object.
When addTaintBySpec or removeTaintBySpec fails (for example due to API server conflicts, RBAC restrictions, or node patch failures), or when node evaluation fails in processAllNodesForRule / processNodeAgainstAllRules, the controller logs the error and increments metrics.Failures, but never calls EventRecorder.Eventf on either the NodeReadinessRule or the affected Node.
As a result:
kubectl describe nodereadinessrule <rule> shows:
Events: <none>
kubectl describe node <node> only shows historical successful events (such as TaintAdded or TaintRemoved) and provides no indication of reconciliation failures.
Operators must rely on controller logs or Prometheus metrics to diagnose scheduling or reconciliation issues.
Steps to Reproduce
Create a NodeReadinessRule with:
enforcementMode: continuous
Induce a failure during addTaintBySpec or removeTaintBySpec (for example by restricting RBAC permissions for patching nodes or forcing an API patch conflict).
Observe that the controller logs an error and increments metrics.Failures.
Run:
kubectl describe nodereadinessrule <rule>
Observe:
Events: <none>
Run:
kubectl get events --field-selector type=Warning
Observe that no Warning event is recorded for the failed reconciliation.
Expected Behavior
When node evaluation or taint operations fail, the controller should emit Kubernetes Warning events following standard controller conventions.
Specifically:
Emit a Warning event on the owning NodeReadinessRule describing the reconciliation failure.
Emit a corresponding Warning event on the affected Node when taint application or removal fails.
Use meaningful event reasons such as AddTaintError, RemoveTaintError, or EvaluationError so operators can diagnose failures directly through kubectl describe.
Controller Version / Image Tag
v0.4.1
Kubernetes Version
v1.36.0 (reproduced on both envtest and kind)
Controller Logs
2026-08-06T09:40:00Z ERROR Failed to evaluate node for rule {"rule":"network-ready","node":"worker-1","error":"failed to remove taint: node \"worker-1\" cannot be patched"}
changed the title [-][BUG] <Short description of the issue>[/-][+][BUG] Controller emits zero Warning events on NodeReadinessRule or Node objects when taint operations fail[/+]on Aug 6, 2026
thanks @ajaysundark , appreciate the feedback. definitely see the issue with event flooding if rbac fails and logs warnings for every node on the rule crd.
to align with #344, i'll update it so node errors (like taint patch failures) only get warning events on the node itself, not the rule. we can limit rule warning events to actual rule-wide failures (like listing nodes or status updates failing).
Activity
[-][BUG] <Short description of the issue>[/-][+][BUG] Controller emits zero Warning events on NodeReadinessRule or Node objects when taint operations fail[/+]ajaysundark commentedon Aug 9, 2026
Can you review #344 and see if this is redundant to the proposed observability shape?
If this was the failure mode, these events may flood the object (rule) possibly hiding other actual events as well.
Sreejesh06 commentedon Aug 9, 2026
thanks @ajaysundark , appreciate the feedback. definitely see the issue with event flooding if rbac fails and logs warnings for every node on the rule crd.
to align with #344, i'll update it so node errors (like taint patch failures) only get warning events on the node itself, not the rule. we can limit rule warning events to actual rule-wide failures (like listing nodes or status updates failing).
pushing these changes soon!