Skip to content

[BUG] Controller emits zero Warning events on NodeReadinessRule or Node objects when taint operations fail #362

Open

Description

@Sreejesh06

What happened?

The controller emits Kubernetes Events for successful taint operations on Node objects, but emits no Warning events when reconciliation or taint operations fail. Additionally, no events are ever emitted on the owning NodeReadinessRule object.

When addTaintBySpec or removeTaintBySpec fails (for example due to API server conflicts, RBAC restrictions, or node patch failures), or when node evaluation fails in processAllNodesForRule / processNodeAgainstAllRules, the controller logs the error and increments metrics.Failures, but never calls EventRecorder.Eventf on either the NodeReadinessRule or the affected Node.

As a result:

  • kubectl describe nodereadinessrule <rule> shows:
    Events: <none>
    
  • kubectl describe node <node> only shows historical successful events (such as TaintAdded or TaintRemoved) and provides no indication of reconciliation failures.
  • Operators must rely on controller logs or Prometheus metrics to diagnose scheduling or reconciliation issues.

Steps to Reproduce

  1. Create a NodeReadinessRule with:
    enforcementMode: continuous
  2. Induce a failure during addTaintBySpec or removeTaintBySpec (for example by restricting RBAC permissions for patching nodes or forcing an API patch conflict).
  3. Observe that the controller logs an error and increments metrics.Failures.
  4. Run:
    kubectl describe nodereadinessrule <rule>
    Observe:
    Events: <none>
    
  5. Run:
    kubectl get events --field-selector type=Warning
    Observe that no Warning event is recorded for the failed reconciliation.

Expected Behavior

When node evaluation or taint operations fail, the controller should emit Kubernetes Warning events following standard controller conventions.

Specifically:

  • Emit a Warning event on the owning NodeReadinessRule describing the reconciliation failure.
  • Emit a corresponding Warning event on the affected Node when taint application or removal fails.
  • Use meaningful event reasons such as AddTaintError, RemoveTaintError, or EvaluationError so operators can diagnose failures directly through kubectl describe.

Controller Version / Image Tag

v0.4.1

Kubernetes Version

v1.36.0 (reproduced on both envtest and kind)

Controller Logs

2026-08-06T09:40:00Z ERROR Failed to evaluate node for rule {"rule":"network-ready","node":"worker-1","error":"failed to remove taint: node \"worker-1\" cannot be patched"}

Additional Environment Details

NIL

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    triage/needs-informationIndicates an issue needs more information in order to work on it.
    on Aug 6, 2026
  2. changed the title [-][BUG] <Short description of the issue>[/-] [+][BUG] Controller emits zero Warning events on NodeReadinessRule or Node objects when taint operations fail[/+] on Aug 6, 2026
  3. ajaysundark commented on Aug 9, 2026

    @ajaysundark
    Contributor

    Can you review #344 and see if this is redundant to the proposed observability shape?

    restricting RBAC permissions for patching nodes

    If this was the failure mode, these events may flood the object (rule) possibly hiding other actual events as well.

  4. Sreejesh06 commented on Aug 9, 2026

    @Sreejesh06
    Author

    thanks @ajaysundark , appreciate the feedback. definitely see the issue with event flooding if rbac fails and logs warnings for every node on the rule crd.

    to align with #344, i'll update it so node errors (like taint patch failures) only get warning events on the node itself, not the rule. we can limit rule warning events to actual rule-wide failures (like listing nodes or status updates failing).

    pushing these changes soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.triage/needs-informationIndicates an issue needs more information in order to work on it.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions