[TRACKING] Update Existing Metrics #446

Summary / Objective

Per the observability design doc #344

Some existing controller metrics need to be renamed, reshaped, or removed to align with the new per-rule metrics. This issue tracks that work.

Rollout plan

• Apply each rename/reshape/deletion separately.
• Keep the old metric running unchanged and add a deprecation note in docs/book/src/operations/monitoring.md pointing to the replacement.
• Publish the replacement in the same release.
• Drop the old metric only after the new one reaches beta (v1beta1)

Checklist / Sub-tasks

• node_readiness_evaluation_duration_seconds - Update node_readiness_evaluation_duration_seconds + tests #447
• node_readiness_rules - feat: add node_readiness_rules with enforcement_mode and dry_run labels #448
• node_readiness_enforcement_latency_seconds - feat: add enforcement latency metric with operation labels #450
• node_readiness_bootstrap_hold_duration_seconds - feat: Add bootstrap NRC duration metric #294
• node_readiness_blocked_nodes - feat: add blocked nodes metric to scrape-time collector #431
• node_readiness_rule_nodes - feat: add scrape-time metrics collector #398
• Monitoring Doc update - docs: update monitoring reference with missing metrics #488

Design Documents & References

Ref. Observability design doc

Evaluation of existing metrics

Existing metric	Decision	Target metric and migration plan
node_readiness_taint_operations_total	Keep	Retain as-is (rule, operation).
node_readiness_bootstrap_completed_total	Keep	Retain as-is (rule).
node_readiness_evaluation_duration_seconds	Reshape	Keep rule label; update help string to clarify it includes taint API calls.
node_readiness_failures_total	Reshape	Migrate reason values to standard snake_case strings (refer Failure reasons).
node_readiness_rules_total	Rename and reshape	Migrate to node_readiness_rules{enforcement_mode, dry_run} gauge. Drop _total suffix from gauge.
node_readiness_nodes_by_state	Rename and reshape	Replace status-derived gauge with an always-on scrape-time collector `node_readiness_rule_nodes{rule, state="held"
node_readiness_bootstrap_duration_seconds	Rename and reshape	Migrate into node_readiness_bootstrap_hold_duration_seconds{rule, taint_origin="adopted"} histogram.
node_readiness_reconciliation_latency_seconds	Rename and reshape	Rename to node_readiness_enforcement_latency_seconds to prevent collision with controller-runtime reconcile metrics. Normalize operation values to add and remove.
node_readiness_condition_failures_total	Delete	Replace state-as-counter metric with the node_readiness_blocked_nodes gauge.
node_readiness_rule_last_reconciliation_timestamp_seconds	Delete	Remove timestamp gauge. Unchanged rules are still actively enforced under GenerationChangedPredicate without resyncs.

We apply all metric renames, label changes, and deletions in one breaking release and document them. We deprecate and dual-publish old and new metric shapes during alpha, and drop the old shapes when graduating to beta.

Target Milestone / Release

No response