Repository navigation
fix(playwright): preserve quarantine across EFD retries - #9892
Conversation
Overall package sizeSelf size: 8.27 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 445.14 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
@codex review |
|
/datadog autotest review |
BenchmarksBenchmark execution time: 2026-08-21 09:38:53 Comparing candidate commit bb8c4c2 in PR branch Found 1 performance improvements and 0 performance regressions! Performance is the same for 2283 metrics, 8 unstable metrics.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f578625a02
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![datadog-ddstaging[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The new EFD integration test starts Playwright outside the automatic cleanup path. A timeout can leave the child process active and can hang the test suite.
🤖 Datadog Autotest · Commit f578625 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
More details
EFD retry copies now keep quarantine data. The code uses the same EFD key for retry status and quarantine status, so quarantined retry failures do not fail the session.
🤖 Datadog Autotest · Commit f578625 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: bb8c4c2 | Docs | View more details | Give us feedback! |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #9892 +/- ##
========================================
Coverage 98.54% 98.55%
========================================
Files 982 983 +1
Lines 145161 145482 +321
Branches 12781 12617 -164
========================================
+ Hits 143055 143379 +324
+ Misses 2106 2103 -3 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
More details
The PR keeps quarantine metadata on Playwright EFD copies and uses the EFD key for failure counts.
The new commit removes order dependence for mixed retry results,
adds a fail-then-pass case,
and adds process cleanup;
no issue remains open.
🤖 Datadog Autotest · Commit 886e166 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 886e1666de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
The PR keeps quarantine data on EFD retry copies and uses one key for retry status. The latest commit also handles a failed first run followed by passing retries. One issue remains: this mixed result can hide an independent beforeAll or afterAll failure and make CI pass.
🤖 Datadog Autotest · Commit 886e166 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
The PR keeps quarantine data on Playwright EFD copies
and uses the same EFD key for failure counts. It fixes the earlier process cleanup issue
and mixed retry result handling, but it keeps a hook error after that hook passes on a native retry;
this open issue can fail a session
when only a quarantined test still fails.
🤖 Datadog Autotest · Commit 4e65c79 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e65c7901a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
The PR preserves quarantine data on EFD copies and uses one key for retry results. It now keeps beforeAll and afterAll failures fatal, but a global teardown failure still bypasses this guard and can make a failed Playwright session pass.
🤖 Datadog Autotest · Commit 4e65c79 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
The PR preserves quarantine data on Playwright EFD copies.
It now counts each EFD failure and handles recovered hooks correctly.
It also keeps global teardown failures visible.
The earlier process cleanup and hook-retry issues are fixed.
One issue remains: custom reporters leave the failure count undefined, so quarantined failures still fail the session.
🤖 Datadog Autotest · Commit f5d81da · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f5d81da057
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
The PR preserves quarantine data on EFD retry copies.
It now keeps suite-hook and global teardown failures fatal.
Multi-project and repeatEach quarantine failures are still undercounted.
Modified EFD clones with native retries are overcounted.
Stdout-only or custom reporters can leave the failure count unset.
🤖 Datadog Autotest · Commit f5d81da · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
@codex review |
|
/datadog autotest review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
Playwright 1.38 ignores failOnFlakyTests, so the new test expects a failure from a run that passes. A second fixture name mismatch leaves the disabled attempt-to-fix case unconfigured.
🤖 Datadog Autotest · Commit e4620f5 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
There was a problem hiding this comment.
A v5 backport leaves the Playwright run summary undefined before version 1.38, so quarantine-only failures fail the session. The oldest test matrix also uses an unsupported failOnFlakyTests option.
🤖 Datadog Autotest · Commit e4620f5 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
@codex review |
|
/datadog autotest review |
There was a problem hiding this comment.
More details
The PR keeps quarantine data on Playwright EFD retry copies.
It uses EFD keys to count quarantine failures and set the session result.
The new commit adds attempt-to-fix data to skipped test spans.
It fixes the Playwright version gate and the fixture name mismatch.
No open finding remains.
🤖 Datadog Autotest · Commit 9ccdbe6 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
This PR keeps quarantine and attempt-to-fix data on Playwright EFD retries and skipped tests. The latest commit fixes the unsupported failOnFlakyTests case and the wrong fixture name, but Playwright 1.18–1.37 still has no reporter summary, so a v5 backport still lets quarantine-only failures fail the session.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · Commit 9ccdbe6 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
Regarding #9892 (review): Playwright 1.18–1.37 is supported by dd-trace v5 for core instrumentation, but not for Test Management or EFD. The capability gates for Playwright quarantine, disable, attempt-to-fix, impacted tests, and EFD are all >=1.38.0 in packages/dd-trace/src/plugins/util/test.js. The Playwright instrumentation also does not request Test Management data below 1.38 (shouldGetTestManagementTests is gated by the same range), and the Test Management integration contexts are gated to 1.38+. Therefore, a 1.18–1.37 run cannot contain Datadog-quarantined tests: it receives no Test Management data and advertises no quarantine capability. A failed session without a reporter summary should remain failed. Adding a legacy quarantine fallback would be a separate expansion of Test Management support below 1.38, not a v5 backport fix for this PR. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9ccdbe672d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
2 participants
What does this PR do?
Preserves Test Management quarantine metadata on Playwright tests cloned for Early Flake Detection retries. It also tracks quarantined failures with the same EFD key used for retry-status aggregation, so an all-failing quarantined test does not fail the session.
Motivation
A quarantined test can also be classified as new or modified because Test Management and EFD use independent inputs. The initial Playwright execution retained
test_management.is_quarantined, but its EFD retry clones did not, allowing those failures to fail the CI process.Additional Notes
Verified with: