Trace aggregation behavior change in agent: nightly failure 2026-09-30 potential root cause. #2723

I need to toss this over the fence because I'm looking at a different issue that is happening right now, and in case I don't get back to this, the investigation and conclusions below seem pretty plausible and actionable.

Nightly runs against datadog-agent main-ish in order to catch integration issues before they land in tagged release candidates. A recent change to trace behavior causes three correctness tests to fail. When @jszwedko re-ran the tests, apparently (according to Opus, below, unverified by me) it no longer ran the same commit from datadog-agent and instead ran with a stable tag (we should also fix that!)

Opus analysis:

Root cause analysis: nightly pipeline 141153981

This is a real integration break (class 2), not a flake. Jesse's rerun passed only because it ran against the stable Agent instead of the nightly one.

There were 2 failing tests, not 3: test-correctness-otlp-traces and test-correctness-otlp-traces-ets. Each failed all 3 auto-retry attempts, in child pipeline 141155097, with the same message every time:

Number of aggregation keys do not match: 119 (baseline) vs 113 (comparison)

1. The Agent change behind the failure

The cause is datadog-agent a458d1f7511, "trace/stats: clamp future span buckets to the current time bucket" (#56608, @ajgajg1134 , merged 09-22). No released tag contains it yet, so it first ships in 7.85.

• Before the change: a span whose end time is in the future lands in a future stats bucket. That bucket isn't flushed until the wall clock reaches it.
• After the change: the bucket is clamped to the current time bucket, so the span is counted and flushed right away.

ADP's port at lib/saluki-components/src/transforms/apm_stats/span_concentrator.rs:329 still has the old logic:

let btime = (end - end % self.bsize).max(self.oldest_ts);

2. Evidence from the failed run's artifacts

The failed run used Agent 7.85.0-devel+git.808.70c956a. The OTLP span timestamps come from lading's random traces generator, and many are centuries in the future.

• All 4096 spans matched between the two sides, apart from the already-ignored meta.http.status_code. The difference is only in stats.
• The 6 extra baseline groups are status-code combinations that only appear among the future-dated spans. Examples: api-gateway GET /api/v1/products 409 and 301, and product-service ListProducts gRPC 6, 12 and 15.
• Only these two tests compare stats. The other four otlp-traces-* cases set otlp_direct_analysis_mode: true, which skips the stats comparison.

3. Why the rerun passed: a race on the image tag

Every scheduled pipeline on the same commit pushes the same mutable tag, bundled-agent-adp:${CI_COMMIT_SHA} (see build-bundled-agent-adp-image in .gitlab/e2e.yml). Three schedules ran on 4cd2f4aa:

The rerun's own logs confirm it: Starting Datadog Agent v7.83.3, plus an ADP build time of 08:03, which belongs to the SMP pipeline's build.

4. The nightly has been failing silently since 09-26

I checked which Agent each recent nightly actually tested:

Every run that got the nightly Agent failed; every recent pass tested a stable Agent. Runs from before 09-24 no longer have artifacts, so I couldn't check them.

Recommended fixes

1. Make the nightly test the Agent it says it tests. The nightly bundled image should use a tag that other pipelines on the same commit can't overwrite. For example, include CI_PIPELINE_ID or a -nightly suffix in the tag, or pass the pushed digest down to the child pipeline. Without this, ~half of nightlies quietly test stable, and retries can "fix" real failures.
2. Bring ADP in line with the new Agent behaviour. Port the clamp into add_span_internal by passing now through. This is also a real memory-retention bug in ADP: future-dated buckets pile up until the clock catches up, which is exactly what the Agent PR fixed.
3. Work around the pinned Agent version. Main pins Agent 7.83.3, so porting the clamp alone would flip the failure onto main (ADP 119 vs Agent 113). The simplest fix that works with both versions is to give these two tests a corpus with no future-dated spans; both Agent behaviours then agree. Clamping into the current bucket instead of dropping only changes the outcome for future-dated spans. Once the pin reaches 7.85 or later, the random timestamps can come back to cover the clamp.