[CONTP-2086][CONTP-2085] Expose cluster_checks metrics via COAT

[adel121]

What does this PR do?

Onboards cluster_check. telemtry via COAT to surface richer cross-org cluster-level health signals.

Onboarded metrics:

Metric Name	Type	Description
cluster_checks.nodes_reporting	Gauge	Number of node agents reporting to the Cluster Agent
endpoint_checks.configs_dispatched	Gauge	Number of endpoint check configurations dispatched
cluster_checks.rebalancing_decisions	Counter	Total number of check rebalancing decisions
cluster_checks.successful_rebalancing_moves	Counter	Total number of successful check rebalancing moves
cluster_checks.failed_stats_collection	Counter	Total number of unsuccessful runner stats collection attempts

The following metrics had already been onboarded:

• cluster_checks.configs_dangling
• cluster_checks.unscheduled_check
• cluster_checks.configs_dispatched
• cluster_checks.configs_info

Explicitly not onboarded (deferred)

Metric	Reason
cluster_checks.busyness	Per-runner value; its only distinguishing label is node (high-card, dropped). Summing weight scores across runners hides per-runner saturation and scales with runner count. Useful fleet signal would be a max, which COAT cannot produce.
cluster_checks.predicted_utilization	Per-runner ratio WorkersUsed/Workers (checks_distribution.go:36-42). Summing ratios is meaningless: 1 runner @1.0 + 9 @0.0 and 10 runners @0.1 both ship 1.0. It cannot distinguish a saturated runner from a healthy fleet.

Explicitly not onboarded (no real value)

Metric	Decision	Reason
cluster_checks.rebalancing_duration_seconds	Not onboarded	Perf/latency, not a health signal; a 15-min flush only spot-samples the last run's duration (no avg/distribution)
cluster_checks.updating_stats_duration_seconds	Not onboarded	Same rationale. The error side of stats collection is already covered by failed_stats_collection.

Describe how you validated your changes

Unit tests.

Manual Testing:

Created kind cluster:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: coat-qa
nodes:
  - role: control-plane
  - role: worker

Created datadog namespace, and created fakeintake:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fakeintake
  namespace: datadog
  labels: {app: fakeintake}
spec:
  replicas: 1
  selector: {matchLabels: {app: fakeintake}}
  template:
    metadata: {labels: {app: fakeintake}}
    spec:
      containers:
        - name: fakeintake
          image: public.ecr.aws/datadog/fakeintake:latest
          imagePullPolicy: IfNotPresent
          ports: [{containerPort: 80}]
---
apiVersion: v1
kind: Service
metadata:
  name: fakeintake
  namespace: datadog
spec:
  selector: {app: fakeintake}
  ports: [{port: 80, targetPort: 80}]

Deployed the agent and created fakeintake service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fakeintake
  namespace: datadog
  labels: {app: fakeintake}
spec:
  replicas: 1
  selector: {matchLabels: {app: fakeintake}}
  template:
    metadata: {labels: {app: fakeintake}}
    spec:
      containers:
        - name: fakeintake
          image: public.ecr.aws/datadog/fakeintake:latest
          imagePullPolicy: IfNotPresent
          ports: [{containerPort: 80}]
---
apiVersion: v1
kind: Service
metadata:
  name: fakeintake
  namespace: datadog
spec:
  selector: {app: fakeintake}
  ports: [{port: 80, targetPort: 80}]
adel.hajhassan@COMP-QH9XH7D725 datadog-agent % cat /tmp/coat-qa-values.yaml 
datadog:
  apiKeyExistingSecret: datadog-secret
  appKeyExistingSecret: datadog-secret
  clusterName: coat-qa
  kubeStateMetricsEnabled: false
  kubeStateMetricsCore:
    enabled: false
  orchestratorExplorer:
    enabled: false
  processAgent:
    enabled: false
  logLevel: INFO

agents:
  enabled: false

clusterAgent:
  enabled: true
  replicas: 1
  admissionController:
    enabled: false
  metricsProvider:
    enabled: false
  clusterChecksEnabled: true
  confd:
    http_check.yaml: |-
      cluster_check: true
      init_config:
      instances:
        - name: k8s-api
          url: https://kubernetes.default.svc
          disable_ssl_validation: true
  datadog_cluster_yaml:
    agent_telemetry:
      enabled: true
      use_http: true
      logs_no_ssl: true
      logs_dd_url: "fakeintake.datadog.svc.cluster.local:80"
      # Same cluster-checks metric set added to defaultProfiles.yaml in this PR,
      # with a short period so a post-warmup flush (non-zero gauges) lands quickly.
      profiles:
        - name: cluster-agent
          metric:
            exclude:
              zero_metric: true
            metrics:
              - name: cluster_checks.configs_dispatched
              - name: cluster_checks.configs_dangling
              - name: cluster_checks.configs_info
                preserve_tags: [check_name]
              - name: cluster_checks.unscheduled_check
                preserve_tags: [config_source]
              - name: cluster_checks.nodes_reporting
              - name: endpoint_checks.configs_dispatched
              - name: cluster_checks.rebalancing_decisions
              - name: cluster_checks.successful_rebalancing_moves
              - name: cluster_checks.failed_stats_collection
          schedule:
            start_after: 45
            iterations: 0
            period: 45

clusterChecksRunner:
  enabled: true
  replicas: 1
  image:
    tag: "7.73.0"

Get pods into running state:

datadog-cluster-agent-...   1/1 Running  
datadog-clusterchecks-...   1/1 Running 
fakeintake-...              1/1 Running

Dispatch confirmed on the DCA:

$ kubectl -n datadog exec <dca> -- agent clusterchecks
=== 1 agents reporting ===
datadog-clusterchecks-...   Running checks: 1   (http_check dispatched)

DCA logs:

comp/core/agenttelemetry/impl/agenttelemetry.go:843 | Starting agent telemetry for 7 schedules and 23 profiles
comp/core/agenttelemetry/impl/agenttelemetry.go:638 | Starting agent telemetry run
agenttelemetry.go:589 | Collect Agent Metric telemetry for profile cluster-agent
agenttelemetry.go:606 | Reporting Agent Metric telemetry for profile cluster-agent

fakeintake received the telemetry on the COAT endpoint:

$ curl .../fakeintake/routestats
{"routes":{"/api/v2/apmtelemetry":{"count":...}}}

Payloads are base64 → zstd → JSON, request_type ∈ {agent-metrics, message-batch}.
The first (embedded, period 900s) flush landed during the dispatcher's 30s warmup, when the
cluster-checks gauges were still 0 and therefore dropped by zero_metric: true. To obtain a
post-warmup flush quickly, the same metric set added in this PR was declared with a short
45s period (still running on the same binary); the embedded wiring itself is covered by the
unit test TestClusterChecksMetricsInClusterAgentProfile.

Additional Notes

• The cluster agent DCA profile excludes zero-values series (excludes zero_metric metrics), cluser_checks.nodes_reporting will not be propagated via COAT when its value is 0. To change this, we will have to create a separate profile for cluser_checks.nodes_reporting so that we can reliably distinguish between "no data available" and "no runners reporting". For now, we accept not shipping the metric via COAT when its value is 0.

[github-actions]

COAT metric reminder

This PR changes comp/core/agenttelemetry/impl/defaultProfiles.yaml, which reports metrics via Cross-Org Agent Telemetry (COAT). Per the COAT: Report a telemetry metric guide, please confirm before merging:

• Validated the metric via e2e tests or a dev VM, and checked it appears correctly in ddcoat (see Observing telemetry — a ready-to-run VM command will be posted in a separate comment once this pipeline's build finishes)
• Updated the public docs in DataDog/documentation
• Filed a new tracking ticket on the ASUP Jira board using the template for the added metric(s): cluster_checks.nodes_reporting, endpoint_checks.configs_dispatched, cluster_checks.rebalancing_decisions, cluster_checks.successful_rebalancing_moves, cluster_checks.failed_stats_collection
• After merge: noted the target Agent release and will confirm GA before considering the metric live

Automated reminder — does not block merging. Updates automatically if this file changes again.

[github-actions]

AI review by Codex (OpenAI) - workflow run

patch is incorrect: the allowlist additions match the metric emitters, but the new documentation claims a health fallback that does not exist. The added test checks configuration only; no E2E assertion verifies delivery of these metrics.

[dd-octo-sts]

COAT metric validation VM

The agent package for this pipeline is ready. Create a VM with that package installed using:

dda inv aws.create-vm --pipeline-id=143021622

Need Windows instead? Add --os-family=windows — but note Windows packages are only
built for this pipeline if it touched a Windows-related path, ran with
RUN_E2E_TESTS/DEPLOY_AGENT, or is on main/a release branch. If yours doesn't
qualify, no Windows build will ever exist for this pipeline id — push an unrelated
Windows-path change, or open a fresh pipeline with RUN_E2E_TESTS=on.

See Observing telemetry for how to find the metric once the agent is running.

[datadog-prod-us1-6]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 62.81% (+4.22%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8c675d0 | Docs | Give us feedback!}

[adel121]

If in the future we find that this limitation is causing issues, we can move this metric to a new separate COAT profile that doesn't exclude zero metrics.

[dd-octo-sts]

Files inventory check summary

File checks results against ancestor ba0496db:

Results for datadog-agent_7.86.0~devel.git.207.8c675d0.pipeline.143021622-1_amd64.deb:

No change detected

Results for datadog-iot-agent_7.86.0~devel.git.207.8c675d0.pipeline.143021622-1_amd64.deb:

No change detected

[dd-octo-sts]

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor ba0496d
📊 Static Quality Gates Dashboard
🔗 SQG Job

Successful checks

Info

	Quality gate	Change	Size (prev → curr → max)
✅	agent_deb_amd64	+4.0 KiB (0.00% increase, -0.07% of buffer)	754.939 → 754.942 → 760.520
✅	agent_heroku_amd64	+4.0 KiB (0.00% increase, -0.52% of buffer)	319.014 → 319.018 → 319.770
✅	agent_rpm_amd64	+4.0 KiB (0.00% increase, -0.07% of buffer)	754.922 → 754.926 → 760.500
✅	agent_rpm_arm64	+8.0 KiB (0.00% increase, -0.14% of buffer)	725.517 → 725.525 → 731.100
✅	agent_rpm_arm64_fips	+4.0 KiB (0.00% increase, -0.07% of buffer)	664.099 → 664.103 → 669.740
✅	agent_suse_amd64	+4.0 KiB (0.00% increase, -0.07% of buffer)	754.922 → 754.926 → 760.500
✅	agent_suse_arm64	+8.0 KiB (0.00% increase, -0.14% of buffer)	725.517 → 725.525 → 731.100
✅	agent_suse_arm64_fips	+4.0 KiB (0.00% increase, -0.07% of buffer)	664.099 → 664.103 → 669.740
✅	docker_agent_amd64	+4.0 KiB (0.00% increase, -0.07% of buffer)	811.113 → 811.117 → 816.720
✅	docker_agent_arm64	+8.0 KiB (0.00% increase, -0.14% of buffer)	806.323 → 806.331 → 811.970
✅	docker_agent_jmx_amd64	+4.01 KiB (0.00% increase, -0.07% of buffer)	1001.983 → 1001.987 → 1007.590
✅	docker_agent_jmx_arm64	+8.0 KiB (0.00% increase, -0.14% of buffer)	985.972 → 985.980 → 991.620
✅	iot_agent_deb_armhf	+4.0 KiB (0.01% increase, -1.01% of buffer)	44.633 → 44.636 → 45.020

19 successful checks with minimal change (< 2 KiB)

	Quality gate	Current Size
✅	agent_deb_amd64_fips	687.964 MiB
✅	agent_rpm_amd64_fips	687.947 MiB
✅	agent_suse_amd64_fips	687.947 MiB
✅	docker_cws_instrumentation_amd64	7.439 MiB
✅	docker_cws_instrumentation_arm64	6.877 MiB
✅	docker_dogstatsd_amd64	39.543 MiB
✅	docker_dogstatsd_arm64	37.629 MiB
✅	docker_host_profiler_amd64	308.675 MiB
✅	docker_host_profiler_arm64	319.791 MiB
✅	dogstatsd_deb_amd64	30.285 MiB
✅	dogstatsd_deb_arm64	28.274 MiB
✅	dogstatsd_rpm_amd64	30.285 MiB
✅	dogstatsd_suse_amd64	30.285 MiB
✅	iot_agent_deb_amd64	47.355 MiB
✅	iot_agent_deb_arm64	43.822 MiB
✅	iot_agent_rpm_amd64	47.356 MiB
✅	iot_agent_suse_amd64	47.355 MiB
✅	docker_cluster_agent_amd64	214.020 MiB
✅	docker_cluster_agent_arm64	226.950 MiB

[adel121]

Docs update PR: DataDog/documentation#40511

[cit-pr-commenter-54b7da]

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Job ID: 88a9d514-6a23-40d5-9442-fc1c217495bc

Baseline: ba0496d
Comparison: 8c675d0
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gate_logs	% cpu utilization	+0.99	[+0.10, +1.88]	1	Logs bounds checks dashboard
➖	quality_gate_metrics_logs	memory utilization	+0.62	[+0.39, +0.84]	1	Logs bounds checks dashboard
➖	quality_gate_private_action_runner	memory utilization	+0.21	[+0.08, +0.34]	1	Logs bounds checks dashboard
➖	dsd_uds_10mb_3k_timestamped_contexts_memory	memory utilization	+0.11	[-0.12, +0.34]	1	Logs
➖	python_openmetrics	% cpu utilization	+0.06	[-0.62, +0.74]	1	Logs bounds checks dashboard
➖	dsd_uds_client_drop_detector_cpu	% cpu utilization	+0.03	[-0.45, +0.52]	1	Logs
➖	quality_gate_security_mean_fs_load	memory utilization	-0.04	[-0.08, +0.01]	1	Logs bounds checks dashboard
➖	quality_gate_idle_all_features	memory utilization	-0.06	[-0.14, +0.01]	1	Logs bounds checks dashboard
➖	quality_gate_idle	memory utilization	-0.13	[-0.17, -0.09]	1	Logs bounds checks dashboard
➖	quality_gate_security_idle	memory utilization	-0.26	[-0.30, -0.22]	1	Logs bounds checks dashboard
➖	quality_gate_security_no_fs_load	memory utilization	-0.29	[-0.37, -0.21]	1	Logs bounds checks dashboard
➖	dsd_uds_10mb_3k_timestamped_contexts_cpu	% cpu utilization	-0.43	[-0.66, -0.20]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	observed_value	links
✅	python_openmetrics	checks_execution_time	10/10	81.40 ≤ 100	bounds checks dashboard
✅	python_openmetrics	cpu_usage	10/10	1341.16 ≤ 1500	bounds checks dashboard
✅	python_openmetrics	memory_usage	10/10	4.31GiB ≤ 4.75GiB	bounds checks dashboard
✅	quality_gate_idle	intake_connections	10/10	4 ≤ 5	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	177.91MiB ≤ 181MiB	bounds checks dashboard
✅	quality_gate_idle	total_bytes_received	10/10	770.23KiB ≤ 819.20KiB	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	2 ≤ 5	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	475.47MiB ≤ 542MiB	bounds checks dashboard
✅	quality_gate_idle_all_features	total_bytes_received	10/10	1.13MiB ≤ 1.25MiB	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	17 ≤ 40	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	215.37MiB ≤ 228MiB	bounds checks dashboard
✅	quality_gate_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_logs	total_bytes_received	10/10	263.89MiB ≤ 292MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	367.04 ≤ 2000	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	19 ≤ 40	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	410.61MiB ≤ 455MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_metrics_logs	total_bytes_received	10/10	0.95GiB ≤ 1.04GiB	bounds checks dashboard
✅	quality_gate_private_action_runner	memory_usage	10/10	73.80MiB ≤ 77MiB	bounds checks dashboard
✅	quality_gate_security_idle	cpu_usage	10/10	29.96 ≤ 100	bounds checks dashboard
✅	quality_gate_security_idle	memory_usage	10/10	328.12MiB ≤ 357MiB	bounds checks dashboard
✅	quality_gate_security_mean_fs_load	cpu_usage	10/10	64.32 ≤ 200	bounds checks dashboard
✅	quality_gate_security_mean_fs_load	memory_usage	10/10	311.61MiB ≤ 337MiB	bounds checks dashboard
✅	quality_gate_security_no_fs_load	cpu_usage	10/10	22.74 ≤ 100	bounds checks dashboard
✅	quality_gate_security_no_fs_load	memory_usage	10/10	315.37MiB ≤ 348MiB	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

• quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
• quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.

[adel121]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-09 10:54:58 UTC ℹ️ Start processing command /merge

---

2026-10-09 10:55:06 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

• Run /code blockers to see what is blocking it.
• Run /remove to cancel it.

---

2026-10-09 14:59:15 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time