Add JUnit support for @Flaky conditions

[sarahchen6]

What Does This Do

Adds JUnit support for @/Flaky conditions and suite restrictions such that -PskipFlakyTests and -PrunFlakyTests work. Now conditions can be added via a String of a method that returns true for the cases where we want to mark the test as flaky.

Motivation

We want to preserve conditional flaky test behavior when migrating Spock tests to JUnit (and when existing JUnit tests become conditionally flaky). Tests should still run normally when the flaky condition does not apply.

Additional Notes

This PR also migrates SampleTraceSmokeTest and CustomSystemLoaderSmokeTest as POCs. See CI results from this PR:

• Latest pipeline that runs non-default JVMs including IBM8: https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/pipelines/141673333
• The tests run normally on non-IBM8 runners: https://app.datadoghq.com/ci/ci-cd/explorer/executions?query=test_level%3Atest%20%40test.service%3Add-trace-java%20-%40ci.job.name%3A%28%22test_smoke%3A%20%5Bibm8%2C%205%2F8%5D%22%20OR%20%22test_ssi_smoke%3A%20%5Bibm8%2C%205%2F8%5D%22%29%20%40test.source.file%3A%28%2ASampleTraceSmokeTest%2A%20OR%20%2ACustomSystemLoaderSmokeTest%2A%29%20%40ci.pipeline.id%3A141673333&agg_m=count&agg_m_source=base&agg_t=count&ci_cd_explorer_tab=tests&cols=%40test.status%2Ctimestamp%2C%40test.suite%2C%40test.name%2C%40duration%2C%40test.service%2C%40git.branch%2C%40test.codeowners%2C%40test.source.file%2C%40ci.job.name&currentTab=overview&eventStack=&fromUser=false&refresh_mode=sliding&start=1790269344003&end=1790874144003&paused=false
• But are skipped for IBM8 where they're marked as Flaky: https://app.datadoghq.com/ci/ci-cd/explorer/executions?query=test_level%3Atest%20%40test.service%3Add-trace-java%20%40ci.job.name%3A%28%22test_smoke%3A%20%5Bibm8%2C%205%2F8%5D%22%20OR%20%22test_ssi_smoke%3A%20%5Bibm8%2C%205%2F8%5D%22%29%20%40test.source.file%3A%28%2ASampleTraceSmokeTest%2A%20OR%20%2ACustomSystemLoaderSmokeTest%2A%29%20%40ci.pipeline.id%3A141673333&agg_m=count&agg_m_source=base&agg_t=count&ci_cd_explorer_tab=tests&cols=%40test.status%2Ctimestamp%2C%40test.suite%2C%40test.name%2C%40duration%2C%40test.service%2C%40git.branch%2C%40test.codeowners%2C%40test.source.file%2C%40ci.job.name&currentTab=overview&eventStack=&fromUser=false&refresh_mode=sliding&start=1790269347595&end=1790874147595&paused=false

which match CI runs from master:

• The tests run normally on non-IBM8 runners: https://app.datadoghq.com/ci/ci-cd/explorer/executions?query=test_level%3Atest%20%40test.service%3Add-trace-java%20-%40ci.job.name%3A%28%22test_smoke%3A%20%5Bibm8%2C%205%2F8%5D%22%20OR%20%22test_ssi_smoke%3A%20%5Bibm8%2C%205%2F8%5D%22%29%20%40test.source.file%3A%28%2ASampleTraceSmokeTest%2A%20OR%20%2ACustomSystemLoaderSmokeTest%2A%29%20%40git.branch%3Amaster&agg_m=count&agg_m_source=base&agg_t=count&ci_cd_explorer_tab=tests&cols=%40test.status%2Ctimestamp%2C%40test.suite%2C%40test.name%2C%40duration%2C%40test.service%2C%40git.branch%2C%40test.codeowners%2C%40test.source.file%2C%40ci.job.name&currentTab=overview&eventStack=&fromUser=false&refresh_mode=sliding&start=1789682137333&end=1790286937333&paused=false
• But are skipped for IBM8 where they're marked as Flaky: https://app.datadoghq.com/ci/ci-cd/explorer/executions?query=test_level%3Atest%20%40test.service%3Add-trace-java%20%40ci.job.name%3A%28%22test_smoke%3A%20%5Bibm8%2C%205%2F8%5D%22%20OR%20%22test_ssi_smoke%3A%20%5Bibm8%2C%205%2F8%5D%22%29%20%40test.source.file%3A%28%2ASampleTraceSmokeTest%2A%20OR%20%2ACustomSystemLoaderSmokeTest%2A%29%20%40git.branch%3Amaster&agg_m=count&agg_m_source=base&agg_t=count&ci_cd_explorer_tab=tests&cols=%40test.status%2Ctimestamp%2C%40test.suite%2C%40test.name%2C%40duration%2C%40test.service%2C%40git.branch%2C%40test.codeowners%2C%40test.source.file%2C%40ci.job.name&currentTab=overview&eventStack=&fromUser=false&refresh_mode=sliding&start=1789682137333&end=1790286937333&paused=false

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.99 s	14.02 s	[-1.0%; +0.5%] (no difference)
startup:insecure-bank:tracing:Agent	12.89 s	13.04 s	[-1.9%; -0.5%] (maybe better)
startup:petclinic:appsec:Agent	17.05 s	17.02 s	[-0.8%; +1.1%] (no difference)
startup:petclinic:iast:Agent	17.15 s	17.13 s	[-0.7%; +1.0%] (no difference)
startup:petclinic:profiling:Agent	16.84 s	16.98 s	[-2.0%; +0.3%] (no difference)
startup:petclinic:sca:Agent	17.13 s	16.73 s	[+1.4%; +3.4%] (significantly worse)
startup:petclinic:tracing:Agent	16.14 s	16.30 s	[-1.9%; -0.1%] (maybe better)

Commit: b7432145 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-24T16:22:09.751106Z	e59ec12	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-24T16:18:44.289774Z	e59ec12	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e59ec123a0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-prod-us1-6]

The most critical issue is that method-level @/Flaky tests are not filtered out before JUnit runs class-level setup and teardown, so smoke-test extensions can still start processes and fail the job even when skipping flaky tests. A separate bug causes the flaky-only run (-PrunFlakyTests) to miss nested tests whose enclosing class carries @/Flaky, because the filter does not walk up the class hierarchy to find the annotation.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit e59ec12 · @DataDog review to ask questions}

[PerfectSlayer]

I left a bunch of comments to answer / review.
Additionally, anything to Flaky (the annotation, JUnit extension and filter) should move to the Java set.

Glad to see progress on the feature! 💪

[PerfectSlayer]

Additional feedback on the changes:

🔨 issue: ‏It seems it always look for class annotation, skipping possible method annotation with different condition.

[sarahchen6]

🔨 issue: ‏It seems it always look for class annotation, skipping possible method annotation with different condition.

The FlakyJUnitExtension should now check method-level -> the test class and its superclasses -> enclosing classes and its superclasses for the Flaky annotation (ref).

[amarziali]

looks ok to me

[bric3]

I'd rather not introduce Stringy condition methods in the Flaky annotation. I believe that relying on existing predicates would already serve the intended purpose. We can eventually bootstrap handy pre-made conditions.

Moreover allowing this stringy condition weakens refactoring awareness like Groovy is. They're harder to navigate to.

As far as I understand this PR was first based on Predicates only, which looks more appropriate to me.

Dismissing myself, I made my statements in the relevant discussions.

[bric3]

I still don't like the JUnit API choice (having a String to link to a method symbol) in this matter. But it's not a blocker either.

[sarahchen6]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-02 15:06:02 UTC ℹ️ Start processing command /merge

---

2026-10-02 15:06:07 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

---

2026-10-02 15:16:21 UTC 🚨 MergeQueue: This merge request is in error because of DDCI

DDCI didn't respond in time.
If this continues, please let us know in #ci-infra-support with the following details:

• Head SHA of the MQ branch: e7835db
• Name of the MQ branch: mq-working-branch-master-78dadd8
• The DDCI Workflow from GenerateDDCIRequestFromDevflow: atlas

[sarahchen6]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-02 15:17:44 UTC ℹ️ Start processing command /merge

---

2026-10-02 15:17:49 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

---

2026-10-02 16:39:45 UTC ℹ️ MergeQueue: This merge request was merged