Skip to content

Add possibility to retry every failed test when retries are enabled #6821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 18, 2024
Merged

Add possibility to retry every failed test when retries are enabled #6821

merged 1 commit into from
Mar 18, 2024

Conversation

@nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Mar 15, 2024 •
edited
Loading

What Does This Do

When flaky test retries are enabled in CI Visibility, by default every failed test will be retried up to N times.
A config property is added to switch the tracer to its previous behaviour: only retry the tests that are known to be flaky.

Jira ticket: CIVIS-9432

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Mar 15, 2024
@pr-commenter
Copy link

pr-commenter bot commented Mar 15, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/flaky-test-retry-everything
git_commit_date 1710526664 1710527121
git_commit_sha 36e924e b1072db
release_version 1.32.0-SNAPSHOT~36e924eaab 1.32.0-SNAPSHOT~b1072db5bc
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1710530594 1710530594
ci_job_id 461342839 461342839
ci_pipeline_id 30217672 30217672
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 2 performance improvements and 2 performance regressions! Performance is the same for 45 metrics, 14 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:tracing:Remote Config better
[-367.546µs; -314.292µs] or [-34.043%; -29.110%]		 0.739ms 1.080ms
scenario:startup:insecure-bank:tracing:Telemetry worse
[+247.043µs; +773.827µs] or [+3.358%; +10.517%]		 7.868ms 7.358ms
scenario:startup:petclinic:tracing:Remote Config better
[-409.854µs; -302.114µs] or [-37.336%; -27.521%]		 0.742ms 1.098ms
scenario:startup:petclinic:tracing:Telemetry worse
[+222.075µs; +699.849µs] or [+3.008%; +9.478%]		 7.845ms 7.384ms
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.086 s) : 0, 1085595
Total [baseline] (9.19 s) : 0, 9189862
Agent [candidate] (1.081 s) : 0, 1080819
Total [candidate] (9.157 s) : 0, 9157496
section appsec
Agent [baseline] (1.199 s) : 0, 1199012
Total [baseline] (9.283 s) : 0, 9283389
Agent [candidate] (1.209 s) : 0, 1209040
Total [candidate] (9.294 s) : 0, 9293520
section iast
Agent [baseline] (1.214 s) : 0, 1214417
Total [baseline] (9.497 s) : 0, 9497147
Agent [candidate] (1.205 s) : 0, 1205197
Total [candidate] (9.302 s) : 0, 9301733
section profiling
Agent [baseline] (1.271 s) : 0, 1271317
Total [baseline] (9.347 s) : 0, 9347314
Agent [candidate] (1.272 s) : 0, 1272473
Total [candidate] (9.401 s) : 0, 9401448
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.086 s -
Agent appsec 1.199 s 113.417 ms (10.4%)
Agent iast 1.214 s 128.823 ms (11.9%)
Agent profiling 1.271 s 185.722 ms (17.1%)
Total tracing 9.19 s -
Total appsec 9.283 s 93.527 ms (1.0%)
Total iast 9.497 s 307.285 ms (3.3%)
Total profiling 9.347 s 157.452 ms (1.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.081 s -
Agent appsec 1.209 s 128.221 ms (11.9%)
Agent iast 1.205 s 124.378 ms (11.5%)
Agent profiling 1.272 s 191.655 ms (17.7%)
Total tracing 9.157 s -
Total appsec 9.294 s 136.024 ms (1.5%)
Total iast 9.302 s 144.237 ms (1.6%)
Total profiling 9.401 s 243.953 ms (2.7%) 
gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (700.331 ms) : 0, 700331
BytebuddyAgent [candidate] (696.225 ms) : 0, 696225
GlobalTracer [baseline] (292.786 ms) : 0, 292786
GlobalTracer [candidate] (292.632 ms) : 0, 292632
AppSec [baseline] (49.427 ms) : 0, 49427
AppSec [candidate] (49.038 ms) : 0, 49038
Remote Config [baseline] (1.098 ms) : 0, 1098
Remote Config [candidate] (741.762 µs) : 0, 742
Telemetry [baseline] (7.384 ms) : 0, 7384
Telemetry [candidate] (7.845 ms) : 0, 7845
section appsec
BytebuddyAgent [baseline] (695.308 ms) : 0, 695308
BytebuddyAgent [candidate] (701.461 ms) : 0, 701461
GlobalTracer [baseline] (290.66 ms) : 0, 290660
GlobalTracer [candidate] (293.563 ms) : 0, 293563
AppSec [baseline] (153.515 ms) : 0, 153515
AppSec [candidate] (153.973 ms) : 0, 153973
IAST [baseline] (17.805 ms) : 0, 17805
IAST [candidate] (18.049 ms) : 0, 18049
Remote Config [baseline] (605.599 µs) : 0, 606
Remote Config [candidate] (609.795 µs) : 0, 610
Telemetry [baseline] (6.846 ms) : 0, 6846
Telemetry [candidate] (6.908 ms) : 0, 6908
section iast
BytebuddyAgent [baseline] (807.269 ms) : 0, 807269
BytebuddyAgent [candidate] (801.104 ms) : 0, 801104
GlobalTracer [baseline] (290.404 ms) : 0, 290404
GlobalTracer [candidate] (289.336 ms) : 0, 289336
AppSec [baseline] (52.453 ms) : 0, 52453
AppSec [candidate] (50.191 ms) : 0, 50191
IAST [baseline] (22.445 ms) : 0, 22445
IAST [candidate] (22.905 ms) : 0, 22905
Remote Config [baseline] (574.706 µs) : 0, 575
Remote Config [candidate] (592.816 µs) : 0, 593
Telemetry [baseline] (6.692 ms) : 0, 6692
Telemetry [candidate] (6.629 ms) : 0, 6629
section profiling
BytebuddyAgent [baseline] (687.795 ms) : 0, 687795
BytebuddyAgent [candidate] (688.09 ms) : 0, 688090
GlobalTracer [baseline] (375.156 ms) : 0, 375156
GlobalTracer [candidate] (376.006 ms) : 0, 376006
AppSec [baseline] (49.83 ms) : 0, 49830
AppSec [candidate] (49.548 ms) : 0, 49548
Remote Config [baseline] (895.934 µs) : 0, 896
Remote Config [candidate] (792.563 µs) : 0, 793
Telemetry [baseline] (7.255 ms) : 0, 7255
Telemetry [candidate] (7.347 ms) : 0, 7347
ProfilingAgent [baseline] (94.263 ms) : 0, 94263
ProfilingAgent [candidate] (94.716 ms) : 0, 94716
Profiling [baseline] (94.286 ms) : 0, 94286
Profiling [candidate] (94.74 ms) : 0, 94740
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.084 s) : 0, 1084186
Total [baseline] (8.559 s) : 0, 8559329
Agent [candidate] (1.083 s) : 0, 1083202
Total [candidate] (8.592 s) : 0, 8591999
section iast
Agent [baseline] (1.202 s) : 0, 1201824
Total [baseline] (9.06 s) : 0, 9059825
Agent [candidate] (1.208 s) : 0, 1207556
Total [candidate] (9.056 s) : 0, 9056304
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.205 s) : 0, 1205141
Total [baseline] (9.018 s) : 0, 9017998
Agent [candidate] (1.213 s) : 0, 1213190
Total [candidate] (9.053 s) : 0, 9052862
section iast_TELEMETRY_OFF
Agent [baseline] (1.198 s) : 0, 1198292
Total [baseline] (9.047 s) : 0, 9046587
Agent [candidate] (1.203 s) : 0, 1203154
Total [candidate] (9.054 s) : 0, 9053899
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.084 s -
Agent iast 1.202 s 117.638 ms (10.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.205 s 120.955 ms (11.2%)
Agent iast_TELEMETRY_OFF 1.198 s 114.106 ms (10.5%)
Total tracing 8.559 s -
Total iast 9.06 s 500.497 ms (5.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.018 s 458.669 ms (5.4%)
Total iast_TELEMETRY_OFF 9.047 s 487.259 ms (5.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.083 s -
Agent iast 1.208 s 124.353 ms (11.5%)
Agent iast_HARDCODED_SECRET_DISABLED 1.213 s 129.988 ms (12.0%)
Agent iast_TELEMETRY_OFF 1.203 s 119.952 ms (11.1%)
Total tracing 8.592 s -
Total iast 9.056 s 464.305 ms (5.4%)
Total iast_HARDCODED_SECRET_DISABLED 9.053 s 460.863 ms (5.4%)
Total iast_TELEMETRY_OFF 9.054 s 461.9 ms (5.4%) 
gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (700.641 ms) : 0, 700641
BytebuddyAgent [candidate] (698.774 ms) : 0, 698774
GlobalTracer [baseline] (291.428 ms) : 0, 291428
GlobalTracer [candidate] (292.315 ms) : 0, 292315
AppSec [baseline] (49.121 ms) : 0, 49121
AppSec [candidate] (49.081 ms) : 0, 49081
Remote Config [baseline] (1.08 ms) : 0, 1080
Remote Config [candidate] (738.744 µs) : 0, 739
Telemetry [baseline] (7.358 ms) : 0, 7358
Telemetry [candidate] (7.868 ms) : 0, 7868
section iast
BytebuddyAgent [baseline] (798.832 ms) : 0, 798832
BytebuddyAgent [candidate] (801.868 ms) : 0, 801868
GlobalTracer [baseline] (287.415 ms) : 0, 287415
GlobalTracer [candidate] (290.069 ms) : 0, 290069
AppSec [baseline] (51.147 ms) : 0, 51147
AppSec [candidate] (52.657 ms) : 0, 52657
IAST [baseline] (22.087 ms) : 0, 22087
IAST [candidate] (21.213 ms) : 0, 21213
Remote Config [baseline] (572.502 µs) : 0, 573
Remote Config [candidate] (628.728 µs) : 0, 629
Telemetry [baseline] (7.402 ms) : 0, 7402
Telemetry [candidate] (6.675 ms) : 0, 6675
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (800.334 ms) : 0, 800334
BytebuddyAgent [candidate] (805.818 ms) : 0, 805818
GlobalTracer [baseline] (288.917 ms) : 0, 288917
GlobalTracer [candidate] (291.091 ms) : 0, 291091
AppSec [baseline] (50.623 ms) : 0, 50623
AppSec [candidate] (51.146 ms) : 0, 51146
IAST [baseline] (23.032 ms) : 0, 23032
IAST [candidate] (22.498 ms) : 0, 22498
Remote Config [baseline] (582.659 µs) : 0, 583
Remote Config [candidate] (613.25 µs) : 0, 613
Telemetry [baseline] (7.29 ms) : 0, 7290
Telemetry [candidate] (7.475 ms) : 0, 7475
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (793.539 ms) : 0, 793539
BytebuddyAgent [candidate] (796.811 ms) : 0, 796811
GlobalTracer [baseline] (289.059 ms) : 0, 289059
GlobalTracer [candidate] (290.958 ms) : 0, 290958
AppSec [baseline] (47.466 ms) : 0, 47466
AppSec [candidate] (51.398 ms) : 0, 51398
IAST [baseline] (25.955 ms) : 0, 25955
IAST [candidate] (22.402 ms) : 0, 22402
Remote Config [baseline] (577.741 µs) : 0, 578
Remote Config [candidate] (566.452 µs) : 0, 566
Telemetry [baseline] (7.325 ms) : 0, 7325
Telemetry [candidate] (6.536 ms) : 0, 6536
Loading

Load

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.361 ms) : 1342, 1380
.   : milestone, 1361,
appsec (1.749 ms) : 1726, 1773
.   : milestone, 1749,
iast (1.52 ms) : 1497, 1543
.   : milestone, 1520,
profiling (1.519 ms) : 1496, 1543
.   : milestone, 1519,
tracing (1.521 ms) : 1498, 1544
.   : milestone, 1521,
section candidate
no_agent (1.358 ms) : 1339, 1377
.   : milestone, 1358,
appsec (1.788 ms) : 1764, 1811
.   : milestone, 1788,
iast (1.499 ms) : 1476, 1522
.   : milestone, 1499,
profiling (1.583 ms) : 1558, 1609
.   : milestone, 1583,
tracing (1.527 ms) : 1505, 1550
.   : milestone, 1527,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.361 ms [1.342 ms, 1.38 ms] -
appsec 1.749 ms [1.726 ms, 1.773 ms] 388.41 µs (28.5%)
iast 1.52 ms [1.497 ms, 1.543 ms] 159.162 µs (11.7%)
profiling 1.519 ms [1.496 ms, 1.543 ms] 158.166 µs (11.6%)
tracing 1.521 ms [1.498 ms, 1.544 ms] 159.95 µs (11.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.358 ms [1.339 ms, 1.377 ms] -
appsec 1.788 ms [1.764 ms, 1.811 ms] 429.759 µs (31.7%)
iast 1.499 ms [1.476 ms, 1.522 ms] 141.142 µs (10.4%)
profiling 1.583 ms [1.558 ms, 1.609 ms] 225.557 µs (16.6%)
tracing 1.527 ms [1.505 ms, 1.55 ms] 169.539 µs (12.5%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~b1072db5bc, baseline=1.32.0-SNAPSHOT~36e924eaab
    dateFormat X
    axisFormat %s
section baseline
no_agent (363.212 µs) : 343, 383
.   : milestone, 363,
iast (469.08 µs) : 449, 489
.   : milestone, 469,
iast_FULL (541.062 µs) : 521, 561
.   : milestone, 541,
iast_GLOBAL (495.0 µs) : 474, 516
.   : milestone, 495,
iast_HARDCODED_SECRET_DISABLED (472.99 µs) : 452, 494
.   : milestone, 473,
iast_INACTIVE (454.503 µs) : 434, 475
.   : milestone, 455,
iast_TELEMETRY_OFF (470.632 µs) : 450, 491
.   : milestone, 471,
tracing (454.708 µs) : 433, 476
.   : milestone, 455,
section candidate
no_agent (366.725 µs) : 347, 386
.   : milestone, 367,
iast (468.521 µs) : 448, 489
.   : milestone, 469,
iast_FULL (533.26 µs) : 513, 554
.   : milestone, 533,
iast_GLOBAL (491.5 µs) : 471, 512
.   : milestone, 492,
iast_HARDCODED_SECRET_DISABLED (476.561 µs) : 456, 497
.   : milestone, 477,
iast_INACTIVE (448.405 µs) : 428, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (472.396 µs) : 452, 493
.   : milestone, 472,
tracing (455.415 µs) : 435, 476
.   : milestone, 455,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 363.212 µs [342.957 µs, 383.467 µs] -
iast 469.08 µs [448.83 µs, 489.33 µs] 105.868 µs (29.1%)
iast_FULL 541.062 µs [520.762 µs, 561.362 µs] 177.85 µs (49.0%)
iast_GLOBAL 495.0 µs [474.44 µs, 515.56 µs] 131.788 µs (36.3%)
iast_HARDCODED_SECRET_DISABLED 472.99 µs [452.156 µs, 493.824 µs] 109.778 µs (30.2%)
iast_INACTIVE 454.503 µs [433.69 µs, 475.316 µs] 91.291 µs (25.1%)
iast_TELEMETRY_OFF 470.632 µs [449.766 µs, 491.498 µs] 107.42 µs (29.6%)
tracing 454.708 µs [433.299 µs, 476.117 µs] 91.496 µs (25.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 366.725 µs [347.048 µs, 386.402 µs] -
iast 468.521 µs [447.956 µs, 489.086 µs] 101.796 µs (27.8%)
iast_FULL 533.26 µs [512.816 µs, 553.704 µs] 166.535 µs (45.4%)
iast_GLOBAL 491.5 µs [471.205 µs, 511.795 µs] 124.775 µs (34.0%)
iast_HARDCODED_SECRET_DISABLED 476.561 µs [455.687 µs, 497.435 µs] 109.836 µs (30.0%)
iast_INACTIVE 448.405 µs [427.509 µs, 469.3 µs] 81.679 µs (22.3%)
iast_TELEMETRY_OFF 472.396 µs [451.504 µs, 493.287 µs] 105.671 µs (28.8%)
tracing 455.415 µs [434.6 µs, 476.23 µs] 88.69 µs (24.2%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review March 16, 2024 11:17
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit b708147 into master Mar 18, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/flaky-test-retry-everything branch March 18, 2024 09:56
@github-actions github-actions bot added this to the 1.32.0 milestone Mar 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anmarchenko anmarchenko anmarchenko approved these changes

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

@dougqh dougqh Awaiting requested review from dougqh dougqh is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility

Projects

None yet

Milestone

1.32.0

Development

Successfully merging this pull request may close these issues.

3 participants

@nikita-tkachenko-datadog @anmarchenko