Increase MavenSmokeTest timeout #9040

daniel-mohedano · 2025-06-26T12:34:10Z

What Does This Do

Increases the Maven smoke test timeout to 20 minutes, due to the tests taking longer after increasing their timeout limit.

Motivation

After the Maven smoke tests were updated to use the Maven proxy for dependencies, they started to time out in CI. Their timeout value was doubled to keep them from failing because of this (#8997). With this change, although the smoke tests don't timeout individually anymore, the :smokeTest target does sometimes go over the timeout limit for Gradle tests configured in configure_tests.gradle (example failure and failed jobs on master)

After some preliminary investigation, it seems like the execution time for the whole job ranges from 10 to 19 minutes.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-06-26T13:09:10Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/increase-maven-smoke-test-timeout
git_commit_date	1750931982	1750943325
git_commit_sha	`0f9b46c`	`f50fa51`
release_version	1.51.0-SNAPSHOT~0f9b46c752	1.51.0-SNAPSHOT~f50fa512c4

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750945267	1750945267
ci_job_id	1000098146	1000098146
ci_pipeline_id	68854166	68854166
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-iun2vzqu-project-304-concurrent-0-69rnfw8w 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-iun2vzqu-project-304-concurrent-0-69rnfw8w 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 45 metrics, 7 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:petclinic:profiling:Remote Config	worse [+14.948µs; +62.647µs] or [+2.291%; +9.600%]	691.369µs	652.571µs

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~f50fa512c4, baseline=1.51.0-SNAPSHOT~0f9b46c752

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (996.216 ms) : 0, 996216
Total [baseline] (10.68 s) : 0, 10680230
Agent [candidate] (996.457 ms) : 0, 996457
Total [candidate] (10.716 s) : 0, 10716285
section appsec
Agent [baseline] (1.177 s) : 0, 1177493
Total [baseline] (10.719 s) : 0, 10719485
Agent [candidate] (1.176 s) : 0, 1176017
Total [candidate] (10.737 s) : 0, 10736570
section iast
Agent [baseline] (1.142 s) : 0, 1141666
Total [baseline] (10.84 s) : 0, 10840009
Agent [candidate] (1.133 s) : 0, 1133278
Total [candidate] (10.852 s) : 0, 10851764
section profiling
Agent [baseline] (1.245 s) : 0, 1244549
Total [baseline] (11.016 s) : 0, 11015675
Agent [candidate] (1.267 s) : 0, 1266702
Total [candidate] (11.13 s) : 0, 11130018

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	996.216 ms	-
Agent	appsec	1.177 s	181.277 ms (18.2%)
Agent	iast	1.142 s	145.451 ms (14.6%)
Agent	profiling	1.245 s	248.334 ms (24.9%)
Total	tracing	10.68 s	-
Total	appsec	10.719 s	39.254 ms (0.4%)
Total	iast	10.84 s	159.779 ms (1.5%)
Total	profiling	11.016 s	335.445 ms (3.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	996.457 ms	-
Agent	appsec	1.176 s	179.56 ms (18.0%)
Agent	iast	1.133 s	136.821 ms (13.7%)
Agent	profiling	1.267 s	270.245 ms (27.1%)
Total	tracing	10.716 s	-
Total	appsec	10.737 s	20.285 ms (0.2%)
Total	iast	10.852 s	135.478 ms (1.3%)
Total	profiling	11.13 s	413.732 ms (3.9%)

gantt
    title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~f50fa512c4, baseline=1.51.0-SNAPSHOT~0f9b46c752

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (687.631 ms) : 0, 687631
BytebuddyAgent [candidate] (687.578 ms) : 0, 687578
GlobalTracer [baseline] (242.574 ms) : 0, 242574
GlobalTracer [candidate] (242.761 ms) : 0, 242761
AppSec [baseline] (30.273 ms) : 0, 30273
AppSec [candidate] (30.369 ms) : 0, 30369
Debugger [baseline] (6.067 ms) : 0, 6067
Debugger [candidate] (6.066 ms) : 0, 6066
Remote Config [baseline] (667.462 µs) : 0, 667
Remote Config [candidate] (666.056 µs) : 0, 666
Telemetry [baseline] (8.238 ms) : 0, 8238
Telemetry [candidate] (8.262 ms) : 0, 8262
section appsec
BytebuddyAgent [baseline] (712.769 ms) : 0, 712769
BytebuddyAgent [candidate] (712.053 ms) : 0, 712053
GlobalTracer [baseline] (236.799 ms) : 0, 236799
GlobalTracer [candidate] (236.016 ms) : 0, 236016
AppSec [baseline] (170.474 ms) : 0, 170474
AppSec [candidate] (170.468 ms) : 0, 170468
Debugger [baseline] (5.786 ms) : 0, 5786
Debugger [candidate] (5.82 ms) : 0, 5820
Remote Config [baseline] (597.159 µs) : 0, 597
Remote Config [candidate] (608.754 µs) : 0, 609
Telemetry [baseline] (8.082 ms) : 0, 8082
Telemetry [candidate] (8.116 ms) : 0, 8116
IAST [baseline] (22.194 ms) : 0, 22194
IAST [candidate] (22.086 ms) : 0, 22086
section iast
BytebuddyAgent [baseline] (815.952 ms) : 0, 815952
BytebuddyAgent [candidate] (809.53 ms) : 0, 809530
GlobalTracer [baseline] (234.339 ms) : 0, 234339
GlobalTracer [candidate] (232.893 ms) : 0, 232893
AppSec [baseline] (28.775 ms) : 0, 28775
AppSec [candidate] (26.842 ms) : 0, 26842
Debugger [baseline] (5.817 ms) : 0, 5817
Debugger [candidate] (5.834 ms) : 0, 5834
Remote Config [baseline] (577.469 µs) : 0, 577
Remote Config [candidate] (578.142 µs) : 0, 578
Telemetry [baseline] (7.969 ms) : 0, 7969
Telemetry [candidate] (7.963 ms) : 0, 7963
IAST [baseline] (27.288 ms) : 0, 27288
IAST [candidate] (28.834 ms) : 0, 28834
section profiling
BytebuddyAgent [baseline] (677.677 ms) : 0, 677677
BytebuddyAgent [candidate] (689.941 ms) : 0, 689941
GlobalTracer [baseline] (361.683 ms) : 0, 361683
GlobalTracer [candidate] (367.412 ms) : 0, 367412
AppSec [baseline] (30.765 ms) : 0, 30765
AppSec [candidate] (35.854 ms) : 0, 35854
Debugger [baseline] (11.348 ms) : 0, 11348
Debugger [candidate] (7.856 ms) : 0, 7856
Remote Config [baseline] (652.571 µs) : 0, 653
Remote Config [candidate] (691.369 µs) : 0, 691
Telemetry [baseline] (10.28 ms) : 0, 10280
Telemetry [candidate] (9.754 ms) : 0, 9754
ProfilingAgent [baseline] (103.439 ms) : 0, 103439
ProfilingAgent [candidate] (105.897 ms) : 0, 105897
Profiling [baseline] (103.463 ms) : 0, 103463
Profiling [candidate] (105.923 ms) : 0, 105923

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~f50fa512c4, baseline=1.51.0-SNAPSHOT~0f9b46c752

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.004 s) : 0, 1004076
Total [baseline] (8.581 s) : 0, 8581132
Agent [candidate] (998.8 ms) : 0, 998800
Total [candidate] (8.594 s) : 0, 8594047
section iast
Agent [baseline] (1.132 s) : 0, 1132199
Total [baseline] (9.289 s) : 0, 9289176
Agent [candidate] (1.129 s) : 0, 1128703
Total [candidate] (9.286 s) : 0, 9286076

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.004 s	-
Agent	iast	1.132 s	128.123 ms (12.8%)
Total	tracing	8.581 s	-
Total	iast	9.289 s	708.044 ms (8.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	998.8 ms	-
Agent	iast	1.129 s	129.903 ms (13.0%)
Total	tracing	8.594 s	-
Total	iast	9.286 s	692.029 ms (8.1%)

gantt
    title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~f50fa512c4, baseline=1.51.0-SNAPSHOT~0f9b46c752

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (693.3 ms) : 0, 693300
BytebuddyAgent [candidate] (688.983 ms) : 0, 688983
GlobalTracer [baseline] (244.127 ms) : 0, 244127
GlobalTracer [candidate] (243.54 ms) : 0, 243540
AppSec [baseline] (30.607 ms) : 0, 30607
AppSec [candidate] (30.412 ms) : 0, 30412
Debugger [baseline] (6.096 ms) : 0, 6096
Debugger [candidate] (6.12 ms) : 0, 6120
Remote Config [baseline] (678.431 µs) : 0, 678
Remote Config [candidate] (675.513 µs) : 0, 676
Telemetry [baseline] (8.321 ms) : 0, 8321
Telemetry [candidate] (8.321 ms) : 0, 8321
section iast
BytebuddyAgent [baseline] (808.778 ms) : 0, 808778
BytebuddyAgent [candidate] (805.822 ms) : 0, 805822
GlobalTracer [baseline] (232.7 ms) : 0, 232700
GlobalTracer [candidate] (232.325 ms) : 0, 232325
AppSec [baseline] (26.093 ms) : 0, 26093
AppSec [candidate] (28.367 ms) : 0, 28367
Debugger [baseline] (5.817 ms) : 0, 5817
Debugger [candidate] (5.781 ms) : 0, 5781
Remote Config [baseline] (593.004 µs) : 0, 593
Remote Config [candidate] (584.56 µs) : 0, 585
Telemetry [baseline] (7.945 ms) : 0, 7945
Telemetry [candidate] (7.966 ms) : 0, 7966
IAST [baseline] (29.535 ms) : 0, 29535
IAST [candidate] (27.141 ms) : 0, 27141

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	daniel.mohedano/increase-maven-smoke-test-timeout
git_commit_date	1750931982	1750943325
git_commit_sha	`0f9b46c`	`f50fa51`
release_version	1.51.0-SNAPSHOT~0f9b46c752	1.51.0-SNAPSHOT~f50fa512c4

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750944906	1750944906
ci_job_id	1000098147	1000098147
ci_pipeline_id	68854166	68854166
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-cpksuxk1-project-304-concurrent-0-94f3z7pj 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-cpksuxk1-project-304-concurrent-0-94f3z7pj 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:no_agent:high_load	better [-208.507µs; -98.235µs] or [-4.679%; -2.204%]	unstable [-72.846op/s; +145.408op/s] or [-7.070%; +14.113%]	4.303ms	1066.594op/s	4.456ms	1030.312op/s
scenario:load:petclinic:tracing:high_load	better [-3.668ms; -2.858ms] or [-8.040%; -6.263%]	unstable [+0.072op/s; +15.678op/s] or [+0.070%; +15.282%]	42.361ms	110.463op/s	45.623ms	102.588op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~f50fa512c4, baseline=1.51.0-SNAPSHOT~0f9b46c752
    dateFormat X
    axisFormat %s
section baseline
no_agent (36.758 ms) : 36455, 37061
.   : milestone, 36758,
appsec (46.962 ms) : 46549, 47375
.   : milestone, 46962,
code_origins (45.047 ms) : 44666, 45428
.   : milestone, 45047,
iast (44.523 ms) : 44130, 44916
.   : milestone, 44523,
profiling (47.259 ms) : 46849, 47669
.   : milestone, 47259,
tracing (45.623 ms) : 45234, 46013
.   : milestone, 45623,
section candidate
no_agent (36.889 ms) : 36593, 37186
.   : milestone, 36889,
appsec (47.058 ms) : 46633, 47483
.   : milestone, 47058,
code_origins (45.839 ms) : 45463, 46215
.   : milestone, 45839,
iast (45.682 ms) : 45294, 46070
.   : milestone, 45682,
profiling (47.972 ms) : 47535, 48409
.   : milestone, 47972,
tracing (42.361 ms) : 41997, 42724
.   : milestone, 42361,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.758 ms [36.455 ms, 37.061 ms]	-
appsec	46.962 ms [46.549 ms, 47.375 ms]	10.204 ms (27.8%)
code_origins	45.047 ms [44.666 ms, 45.428 ms]	8.289 ms (22.6%)
iast	44.523 ms [44.13 ms, 44.916 ms]	7.765 ms (21.1%)
profiling	47.259 ms [46.849 ms, 47.669 ms]	10.501 ms (28.6%)
tracing	45.623 ms [45.234 ms, 46.013 ms]	8.865 ms (24.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.889 ms [36.593 ms, 37.186 ms]	-
appsec	47.058 ms [46.633 ms, 47.483 ms]	10.169 ms (27.6%)
code_origins	45.839 ms [45.463 ms, 46.215 ms]	8.949 ms (24.3%)
iast	45.682 ms [45.294 ms, 46.07 ms]	8.793 ms (23.8%)
profiling	47.972 ms [47.535 ms, 48.409 ms]	11.083 ms (30.0%)
tracing	42.361 ms [41.997 ms, 42.724 ms]	5.471 ms (14.8%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~f50fa512c4, baseline=1.51.0-SNAPSHOT~0f9b46c752
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.456 ms) : 4403, 4510
.   : milestone, 4456,
iast (9.217 ms) : 9070, 9365
.   : milestone, 9217,
iast_FULL (14.089 ms) : 13808, 14370
.   : milestone, 14089,
iast_GLOBAL (10.376 ms) : 10196, 10557
.   : milestone, 10376,
profiling (8.632 ms) : 8490, 8774
.   : milestone, 8632,
tracing (7.786 ms) : 7668, 7903
.   : milestone, 7786,
section candidate
no_agent (4.303 ms) : 4254, 4352
.   : milestone, 4303,
iast (9.384 ms) : 9223, 9545
.   : milestone, 9384,
iast_FULL (13.652 ms) : 13385, 13918
.   : milestone, 13652,
iast_GLOBAL (10.406 ms) : 10219, 10593
.   : milestone, 10406,
profiling (8.876 ms) : 8730, 9022
.   : milestone, 8876,
tracing (7.987 ms) : 7863, 8112
.   : milestone, 7987,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.456 ms [4.403 ms, 4.51 ms]	-
iast	9.217 ms [9.07 ms, 9.365 ms]	4.761 ms (106.8%)
iast_FULL	14.089 ms [13.808 ms, 14.37 ms]	9.633 ms (216.2%)
iast_GLOBAL	10.376 ms [10.196 ms, 10.557 ms]	5.92 ms (132.8%)
profiling	8.632 ms [8.49 ms, 8.774 ms]	4.176 ms (93.7%)
tracing	7.786 ms [7.668 ms, 7.903 ms]	3.329 ms (74.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.303 ms [4.254 ms, 4.352 ms]	-
iast	9.384 ms [9.223 ms, 9.545 ms]	5.081 ms (118.1%)
iast_FULL	13.652 ms [13.385 ms, 13.918 ms]	9.349 ms (217.3%)
iast_GLOBAL	10.406 ms [10.219 ms, 10.593 ms]	6.103 ms (141.8%)
profiling	8.876 ms [8.73 ms, 9.022 ms]	4.573 ms (106.3%)
tracing	7.987 ms [7.863 ms, 8.112 ms]	3.684 ms (85.6%)

Dacapo

increase maven smoke test timeout to 20

6d40285

daniel-mohedano added the tag: no release notes Changes to exclude from release notes label Jun 26, 2025

use chronounit, mirroring gradle smoke test configuration

f50fa51

daniel-mohedano changed the title ~~Increase Maven Smoke Test timeout~~ Increase smoke test timeout Jun 26, 2025

daniel-mohedano changed the title ~~Increase smoke test timeout~~ Increase MavenSmokeTest timeout Jun 26, 2025

daniel-mohedano marked this pull request as ready for review June 26, 2025 14:35

daniel-mohedano requested a review from a team as a code owner June 26, 2025 14:35

daniel-mohedano requested a review from Mariovido June 26, 2025 14:35

AlexeyKuznetsov-DD approved these changes Jun 27, 2025

View reviewed changes

PerfectSlayer added type: bug Bug report and fix comp: tooling Build & Tooling labels Jun 27, 2025

daniel-mohedano merged commit bc8bbe2 into master Jun 27, 2025
519 of 520 checks passed

daniel-mohedano deleted the daniel.mohedano/increase-maven-smoke-test-timeout branch June 27, 2025 08:14

github-actions bot added this to the 1.51.0 milestone Jun 27, 2025

daniel-mohedano mentioned this pull request Jun 30, 2025

Increase CiVis smoke tests timeout #9062

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Increase MavenSmokeTest timeout #9040

Increase MavenSmokeTest timeout #9040

Uh oh!

daniel-mohedano commented Jun 26, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jun 26, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Increase MavenSmokeTest timeout #9040

Increase MavenSmokeTest timeout #9040

Uh oh!

Conversation

daniel-mohedano commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

daniel-mohedano commented Jun 26, 2025 •

edited

Loading

pr-commenter bot commented Jun 26, 2025 •

edited

Loading