Skip to content

[system_tests] Fix flaky test_app_started_is_first_message by using seq_id ordering#6387

Open
vlad-scherbich wants to merge 1 commit intomainfrom
vlad/system-tests-deflake-test_app_started_is_first_message
Open

[system_tests] Fix flaky test_app_started_is_first_message by using seq_id ordering#6387
vlad-scherbich wants to merge 1 commit intomainfrom
vlad/system-tests-deflake-test_app_started_is_first_message

Conversation

@vlad-scherbich
Copy link
Contributor

@vlad-scherbich vlad-scherbich commented Feb 26, 2026

https://datadoghq.atlassian.net/browse/PROF-13796

Motivation

test_telemetry.py::Test_Telemetry::test_app_started_is_first_message was flaky (e.g. 8 hits in dd-trace-py main CI, 2026). The test asserted that app-started is the first telemetry message, but it used capture order (log file order) instead of send order (seq_id). Capture order can be non-deterministic—when the first captured batch is not the first sent, the test fails.

Changes

  • test_app_started_is_first_message: Use seq_id as the source of truth for ordering. Flatten all lifecycle telemetry (single messages + message-batch payloads), skip non-lifecycle types (sketches, generate-metrics, logs, distributions), group by runtime_id, sort by (seq_id, batch_index), and assert per runtime that the first message is app-started. This removes dependence on capture/file order.

Reviewer checklist

  • Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
  • A docker base image is modified?
    • the relevant build-XXX-image label is present
  • A scenario is added, removed or renamed?

@github-actions
Copy link
Contributor

CODEOWNERS have been resolved as:

tests/test_telemetry.py                                                 @DataDog/libdatadog-telemetry @DataDog/apm-sdk-capabilities @DataDog/system-tests-core

@vlad-scherbich vlad-scherbich changed the title Fix flakiness by relying on send order (seq_id) vs capture order (log… [system_tests] Fix flaky test_app_started_is_first_message by using seq_id ordering Feb 27, 2026
@vlad-scherbich vlad-scherbich force-pushed the vlad/system-tests-deflake-test_app_started_is_first_message branch 4 times, most recently from eba115d to c14cbfb Compare February 27, 2026 15:49
@vlad-scherbich vlad-scherbich force-pushed the vlad/system-tests-deflake-test_app_started_is_first_message branch from c14cbfb to a418fd7 Compare February 27, 2026 16:05
@vlad-scherbich vlad-scherbich marked this pull request as ready for review February 27, 2026 16:32
@vlad-scherbich vlad-scherbich requested review from a team as code owners February 27, 2026 16:32
@vlad-scherbich vlad-scherbich requested review from a team, KowalskiThomas, pawelchcki and rachelyangdog and removed request for a team February 27, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant