Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions comp/core/agenttelemetry/impl/agenttelemetry_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2982,6 +2982,52 @@ func TestInstrumentationControllerMetricsInClusterAgentProfile(t *testing.T) {
assert.ElementsMatch(t, []string{"section", "status"}, metrics["instrumentation_controller.reconciliations"])
}

// TestClusterChecksMetricsInClusterAgentProfile guards the cluster-checks / CLC telemetry
// onboarded to COAT. A metric missing from the allowlist is dropped rather than shipped.
//
// These metrics are intentionally emitter-only: their only distinguishing label is the
// per-node (CLC runner) label, which is high-cardinality and must be dropped, so COAT sums
// them into meaningful cluster-wide counts/totals (configs dispatched, nodes reporting,
// rebalancing decisions/moves, failed stats collections). busyness and predicted_utilization
// are deliberately NOT onboarded: they are per-runner values (a ratio, for utilization) whose
// additive aggregation across runners is meaningless, so they must stay out of this profile.
func TestClusterChecksMetricsInClusterAgentProfile(t *testing.T) {
cfg, err := parseConfig(configmock.NewFromYAML(t, defaultProfiles))
require.NoError(t, err)

var profile *Profile
for _, candidate := range cfg.Profiles {
if candidate.Name == "cluster-agent" {
profile = candidate
break
}
}
require.NotNil(t, profile)
require.NotNil(t, profile.Metric)

metrics := make(map[string][]string, len(profile.Metric.Metrics))
for _, metric := range profile.Metric.Metrics {
metrics[metric.Name] = metric.PreserveTags
}

// Onboarded, emitter-only (no preserve_tags).
for _, name := range []string{
"cluster_checks.nodes_reporting",
"endpoint_checks.configs_dispatched",
"cluster_checks.rebalancing_decisions",
"cluster_checks.successful_rebalancing_moves",
"cluster_checks.failed_stats_collection",
} {
preserved, ok := metrics[name]
require.Truef(t, ok, "%s is not allowlisted, so it would never be sent", name)
assert.Emptyf(t, preserved, "%s must be emitter-only (no preserve_tags)", name)
}

// Deliberately excluded: per-runner values whose additive aggregation is meaningless.
assert.NotContains(t, metrics, "cluster_checks.busyness")
assert.NotContains(t, metrics, "cluster_checks.predicted_utilization")
}

// TestDataPlanePreflightModeProfile guards the Agent Data Plane preflight mode metrics.
//
// The pre-flight in comp/dataplane/preflightmode reports its outcome purely through these
Expand Down
14 changes: 14 additions & 0 deletions comp/core/agenttelemetry/impl/defaultProfiles.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,20 @@ profiles:
- name: cluster_checks.unscheduled_check
preserve_tags:
- config_source
# cluster checks: node reporting and endpoint-check dispatch. These carry only
# a per-node label (dropped, high-cardinality) so they aggregate to emitter-only
# counts/totals.
# Accepted limitation: zero_metric exclusion (below) drops nodes_reporting==0, so the
# "no runners reporting" state produces no signal. Only positive counts are observable.
- name: cluster_checks.nodes_reporting

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If in the future we find that this limitation is causing issues, we can move this metric to a new separate COAT profile that doesn't exclude zero metrics.

- name: endpoint_checks.configs_dispatched
# cluster checks: rebalancing activity (additive counters). Sustained high rates
# indicate an unstable or churning check distribution.
- name: cluster_checks.rebalancing_decisions
- name: cluster_checks.successful_rebalancing_moves
# cluster checks: stats-collection health. failed_stats_collection is a direct
# error signal (DCA cannot collect stats from a runner).
- name: cluster_checks.failed_stats_collection
- name: language_detection_patcher.patches
preserve_tags:
- owner_kind
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
other:
- |
Added Cross-Org Agent Telemetry (COAT) metrics to the ``cluster-agent`` profile to
surface cluster checks and Cluster Level Check (CLC) runner health fleet-wide. The
metrics are emitter-only (the per-node label is aggregated away):

| Metric Name | Type | Description |
| --- | --- | --- |
| ``cluster_checks.nodes_reporting`` | Gauge | Number of node agents reporting to the Cluster Agent |
| ``endpoint_checks.configs_dispatched`` | Gauge | Number of endpoint check configurations dispatched |
| ``cluster_checks.rebalancing_decisions`` | Counter | Total number of check rebalancing decisions |
| ``cluster_checks.successful_rebalancing_moves`` | Counter | Total number of successful check rebalancing moves |
| ``cluster_checks.failed_stats_collection`` | Counter | Total number of unsuccessful runner stats collection attempts |
Loading