Repository navigation
[FEATURE] [LFX] Headlamp plugin for node-readiness-controller #327
Description
Is your feature request related to a problem or existing issue? Please describe.
NRC state today is only visible via kubectl, logs, and the Prometheus metrics surface. It may be desirable for the Headlamp users to get answers to "why is this node not accepting workloads?" directly from the UX.
This issue tracks building a plugin for Headlamp to visualize NRC state directly, proposed as a CNCF LFX mentorship project (2026 Term 3). This is a follow-up deliverable from the Term 2 observability project.
Describe the solution you'd like
High-level scope:
- Cluster-level view of NodeReadinessRule objects with live status (nodes matched / held / released per rule, derived from node taints).
- Node-level drill-down: applicable rules, per-condition pass/fail with failure reasons, NRC-managed taints, and related Events.
- Watch-based, API-state only: This doesnt replace the metrics / Grafana. Headlamp will give visibility into per-object questions and capture relationships on the map.
- Align with other standardized node conditions to explore as a generic lifecycle-conditions panel for Node Availability.
- Design doc reviewed by NRC and Headlamp maintainers before implementation.
- Validate and document the UX and scale tests / limits for the plugin.
- Published to Headlamp Artifact Hub with docs.
Describe alternatives you've considered
No response
Activity
ajaysundark commentedon Jul 28, 2026
This for LFX Term3 mentorship project, please refrain from self-assigning the feature.
cc @AvineshTripathi @AnuragThePathak @illume
[-][FEATURE] Headlamp plugin for node-readiness-controller (LFX mentorship project)[/-][+][FEATURE] [LFX] Headlamp plugin for node-readiness-controller[/+]soumadipcodeEnthusiast commentedon Aug 5, 2026
Thanks for opening this — not self-assigning, just a couple of scoping questions ahead of Term 3 applications:
On generalizing to other node conditions beyond NRC — should the initial build stay NRC-specific (NodeReadinessRule + managed taints), with the generic lifecycle-conditions panel as a stretch goal? Or generic from day one?
Since this follows the Term 2 observability project — is there an existing design doc or plugin skeleton from that term worth reading first, or is this a clean start?
LarytheLord commentedon Aug 6, 2026
I have been reading toward this and two open PRs change what the plugin should be built on, so sharing before I write anything longer.
#344 says per-node lists are being removed from
NodeReadinessRule.statusfor scalability, and that per-node drill-downs move to the newNodeReadinessEvaluationCRD in #345. That matters here because the natural way to build the node view, listing rules and readingappliedNodesandfailedNodes, would be building on something you are deleting. So the data sources look like they split by scope rather than by screen:NodeReadinessRule.statusNodeReadinessEvaluationget, which replaces a client-side join across every ruleTaintAdded,TaintRemovedandTaintAdoptedare emitted against the Node today innode_controller.goandnodereadinessrule_controller.go, and docs: add observability design #344 addsBootstrapCompleted, so the drill-down needs no controller changes to show historyThe NRE timestamps also answer time in state directly rather than by inference.
firstEvaluatedAt,taintObservedAt,taintAddedAtandtaintRemovedAtgive hold duration per rule per node, which is the thing an operator actually asks when a node is not accepting workloads.My question is about the feature flag. NRE is opt-in behind
--enable-node-readiness-evaluation. Does the plugin require it, or does it degrade when the flag is off? Requiring it keeps the plugin simple and matches where the project is going. Degrading means keeping a fallback path that reconstructs per-node state from rule status, which is the code you are removing, so it would be carrying the old model forward inside the UI. I lean towards requiring NRE and showing an explicit empty state that names the flag, but that is a product call rather than a technical one.On NRC-specific versus the generic lifecycle-conditions panel, I would build NRC-specific first with the condition reading behind one interface, so generic becomes a swap rather than a rewrite. #344 noting that
kube_node_status_conditionalready exports custom conditions through kube-state-metrics suggests the generic view has a data source that does not depend on NRC at all, which supports keeping the two separate from the start.DsThakurRawat commentedon Aug 7, 2026
Not self-assigning — a datasource question that I think decides the plugin's architecture, and I could not find it answered in #344 or the thread.
#344 routes every fleet rollup to Prometheus:
node_readiness_rule_nodes{rule,state},node_readiness_rule_matched_nodesandnode_readiness_blocked_nodesare all served by the new scrape-time collector, explicitly because per-node lists are leavingNodeReadinessRule.status. That is clearly right for a Grafana SLO view.But a Headlamp plugin runs in the browser and reaches the cluster through Headlamp's API proxy. In the Headlamp tree,
frontend/src/lib/k8s/api/v1/metricsApi.tsfetches metrics withclusterRequestagainst/apis/metrics.k8s.io/v1beta1/...— an aggregated API served by the apiserver, not a scrape of a controller's/metrics. The in-treeplugins/examples/resource-chartsbuilds its rollup client-side fromK8s.ResourceClasses.Pod.useList(). The only Prometheus integration I can find is a catalog-installed plugin rather than something core, so a plugin cannot assume a Prometheus datasource is present or reachable.That leaves two branches, and they lead to different plugins:
NodeReadinessRule.statusis what docs: add observability design #344 removes, and rebuilding it fromNodeReadinessEvaluationmeans a LIST across every node — the client-side join ✨ feat: introduce NodeReadinessEvaluation (NRE) CRD and controller #345 is meant to eliminate.So my question is which one you intend. A third reading is that the split is deliberate: Grafana owns the fleet SLO view under #182, and the plugin owns what the API can answer well — per-rule health, per-node drill-down from a single NRE get, and lifecycle from the Node events that already exist. If that is the intent, it would be worth stating in #327, because both branches above are reasonable to build toward and they produce different plugins.
This also compounds the feature-flag question above:
NodeReadinessEvaluationis opt-in behind--enable-node-readiness-evaluation, so a cluster can be missing the per-node source and the fleet source independently. Knowing which of those the plugin is allowed to require would settle the empty states before any code exists.LarytheLord commentedon Aug 10, 2026
@DsThakurRawat I went and read the observability design doc itself rather than just the PR
summary, and think it answers part of this. Its Goals are "add granular per-rule metrics"
and "build a Grafana dashboard for fleet-wide SLOs," and its Non-Goals say plainly: "Build a
Headlamp plugin. This was originally a goal, but we rescoped it as a follow-up project for
LFX Term 3." If that reading is right, the plugin would not need Prometheus for the fleet-SLO
view specifically, since that view is explicitly someone else's scope, and per-node would
come from a single
NodeReadinessEvaluationget either way.Where I am less sure is per-rule counts without Prometheus.
NodeReadinessRuleStatusonmainstill carriesappliedNodesandfailedNodestoday, but I would call both shakyrather than free right now: #341 (open, reproduced) shows
updateRuleStatuscan silentlylose concurrent per-node writes to either field with no error or retry, and #381 (open) shows
failedNodesseparately getting stuck with stale entries that never clear on their own. Thedesign doc also proposes removing both once the scrape-time collector ships, so today's
counts are not just scheduled to go away, they are not fully trustworthy yet either. One
small thing worth knowing while reading #345: its
NodeReadinessEvaluationReconcilerisdeliberately only eventually consistent with the Node it derives from (per Karthik-K-N's
reply this morning), so a per-node NRE read can briefly lag the Node object it is summarizing.
Not weighing in on that design, just flagging it in case the plugin side had not seen it yet.
@ajaysundark, the question underneath all of this is one only you can really settle. v0.5.0
is cut August 14th per #377, and its changelog includes neither #344 nor #345, and #345
itself is still being discussed as of this morning. So whoever starts this in September is
choosing between two different twelve weeks: build the plugin's first weeks around helping
land NRE and the metrics work, or scope entirely against what exists on
maintoday(
appliedNodes/failedNodes, no per-node CRD) and treat NRE as an enhancement to bolt ononce it ships. Which of those is closer to how you would want this scoped, or is that itself
still undecided on your side?
DsThakurRawat commentedon Aug 10, 2026
@LarytheLord one correction on the second of the two counts you called shaky: #381 is the fix, not the symptom. the stale
failedNodesbug it addresses is #376, and i ran the pr branch today (873d145, which includes the assertion fix from this afternoon) against that repro: the stale entry clears from the persisted rule status and the full suite is green. so that half is a fix pending review rather than an open problem. can't speak to #341, i haven't tried its repro.tejassinghbhati commentedon Aug 13, 2026
I guess that there is one thing on using Node events for the lifecycle history.
Worth knowing that
TaintAdded/TaintRemoved/TaintAdoptedwere not actually being recorded on Helm installs until #351 landed last week. The chart's ClusterRole was missing the events.k8s.io group, so every one of those writes came back forbidden and nothing showed up. On any cluster installed from the chart before that, the history the plugin would read is just empty.The bigger thing is retention. Events default to around an hour TTL, so they answer "what happened recently" but not "how long has this node been held". The NRE timestamps are the better source for that, and @ajaysundark 's own point on #384 was that events are best-effort by design, so I would not lean on them as the history store.
feat(lab): measure each candidate plugin datasource
AnuragThePathak commentedon Sep 15, 2026
As things stand, this are the planned requirements for the Headlamp plugin within this term. The document may undergo some minor changes as discussions progress, which will be updated in document itself https://docs.google.com/document/d/1YMRIF5EAeeSm2aoRtU7bznTyW6i2mmBkd123kku2F_M/edit?usp=sharing
cc @ajaysundark @illume @vishnukothakapu
/assign @vishnukothakapu
kubernetes-prow commentedon Sep 15, 2026
@AnuragThePathak: GitHub didn't allow me to assign the following users: vishnukothakapu.
Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
vishnukothakapu commentedon Sep 16, 2026
/assign
node-readiness-controller: Add initial plugin project structure
node-readiness-controller: Add initial plugin project structure
node-readiness-controller: Add initial plugin project structure
node-readiness-controller: Add initial plugin project structure