Skip to content

Raised relativeAccuracy to 0.2 since 0.1 causes ~16% random failures at n=10,000 due to expected stddev fluctuations. #9588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 23, 2025
Merged

Raised relativeAccuracy to 0.2 since 0.1 causes ~16% random failures at n=10,000 due to expected stddev fluctuations. #9588

merged 1 commit into from
Sep 23, 2025

Conversation

@AlexeyKuznetsov-DD
Copy link
Contributor

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD commented Sep 22, 2025 •
edited
Loading

What Does This Do

Raised relativeAccuracy to 0.2 since 0.1 causes ~16% random failures at n=10,000 due to expected stddev fluctuations.

Motivation

Green CI.

Additional Notes

Fixed flaky test that failing with 16% probability.

@AlexeyKuznetsov-DD
Raised relativeAccuracy to 0.2 since 0.1 causes `~16% random …
acd784f 
…failures at `n=10,000` due to expected `stddev` fluctuations.
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD self-assigned this Sep 22, 2025
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Sep 22, 2025
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD marked this pull request as ready for review September 22, 2025 20:45
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD requested a review from a team as a code owner September 22, 2025 20:45
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD changed the title Raised relativeAccuracy to 0.2 since 0.1 causes `~16% random failures at n=10,000 due to expected stddev fluctuations. Raised relativeAccuracy to 0.2 since 0.1 causes ~16% random failures at n=10,000 due to expected stddev fluctuations. Sep 22, 2025
@datadog-official
Copy link

datadog-official bot commented Sep 22, 2025

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 80.34% (+20.51%)

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: acd784f | Docs | Was this helpful? Give us feedback!

@pr-commenter
Copy link

pr-commenter bot commented Sep 22, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master alexeyk/fixed-flaky-histogram-test
git_commit_date 1758571586 1758573786
git_commit_sha 7b1d89d acd784f
release_version 1.54.0-SNAPSHOT~7b1d89d384 1.54.0-SNAPSHOT~acd784fb1b
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1758575706 1758575706
ci_job_id 1142107322 1142107322
ci_pipeline_id 77223452 77223452
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-ij1fcn76 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-ij1fcn76 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 8 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.017 s) : 0, 1017408
Total [baseline] (10.769 s) : 0, 10768769
Agent [candidate] (1.024 s) : 0, 1023972
Total [candidate] (10.775 s) : 0, 10774979
section appsec
Agent [baseline] (1.194 s) : 0, 1194197
Total [baseline] (11.012 s) : 0, 11011837
Agent [candidate] (1.194 s) : 0, 1194402
Total [candidate] (10.938 s) : 0, 10937943
section iast
Agent [baseline] (1.154 s) : 0, 1154148
Total [baseline] (11.028 s) : 0, 11028120
Agent [candidate] (1.152 s) : 0, 1151990
Total [candidate] (11.117 s) : 0, 11117347
section profiling
Agent [baseline] (1.164 s) : 0, 1164463
Total [baseline] (11.038 s) : 0, 11037824
Agent [candidate] (1.162 s) : 0, 1161904
Total [candidate] (11.053 s) : 0, 11053002
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.017 s -
Agent appsec 1.194 s 176.789 ms (17.4%)
Agent iast 1.154 s 136.74 ms (13.4%)
Agent profiling 1.164 s 147.055 ms (14.5%)
Total tracing 10.769 s -
Total appsec 11.012 s 243.067 ms (2.3%)
Total iast 11.028 s 259.351 ms (2.4%)
Total profiling 11.038 s 269.055 ms (2.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.024 s -
Agent appsec 1.194 s 170.43 ms (16.6%)
Agent iast 1.152 s 128.018 ms (12.5%)
Agent profiling 1.162 s 137.932 ms (13.5%)
Total tracing 10.775 s -
Total appsec 10.938 s 162.964 ms (1.5%)
Total iast 11.117 s 342.368 ms (3.2%)
Total profiling 11.053 s 278.023 ms (2.6%) 
gantt
    title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.453 ms) : 0, 1453
crashtracking [candidate] (1.454 ms) : 0, 1454
BytebuddyAgent [baseline] (686.254 ms) : 0, 686254
BytebuddyAgent [candidate] (690.914 ms) : 0, 690914
GlobalTracer [baseline] (257.415 ms) : 0, 257415
GlobalTracer [candidate] (258.957 ms) : 0, 258957
AppSec [baseline] (31.567 ms) : 0, 31567
AppSec [candidate] (31.845 ms) : 0, 31845
Debugger [baseline] (6.37 ms) : 0, 6370
Debugger [candidate] (6.386 ms) : 0, 6386
Remote Config [baseline] (687.066 µs) : 0, 687
Remote Config [candidate] (681.662 µs) : 0, 682
Telemetry [baseline] (12.652 ms) : 0, 12652
Telemetry [candidate] (12.619 ms) : 0, 12619
section appsec
crashtracking [baseline] (1.455 ms) : 0, 1455
crashtracking [candidate] (1.45 ms) : 0, 1450
BytebuddyAgent [baseline] (709.473 ms) : 0, 709473
BytebuddyAgent [candidate] (709.345 ms) : 0, 709345
GlobalTracer [baseline] (249.156 ms) : 0, 249156
GlobalTracer [candidate] (249.754 ms) : 0, 249754
AppSec [baseline] (171.207 ms) : 0, 171207
AppSec [candidate] (171.143 ms) : 0, 171143
Debugger [baseline] (6.065 ms) : 0, 6065
Debugger [candidate] (6.027 ms) : 0, 6027
Remote Config [baseline] (624.564 µs) : 0, 625
Remote Config [candidate] (610.9 µs) : 0, 611
Telemetry [baseline] (9.949 ms) : 0, 9949
Telemetry [candidate] (9.88 ms) : 0, 9880
IAST [baseline] (25.093 ms) : 0, 25093
IAST [candidate] (25.075 ms) : 0, 25075
section iast
crashtracking [baseline] (1.455 ms) : 0, 1455
crashtracking [candidate] (1.456 ms) : 0, 1456
BytebuddyAgent [baseline] (808.711 ms) : 0, 808711
BytebuddyAgent [candidate] (807.473 ms) : 0, 807473
GlobalTracer [baseline] (248.412 ms) : 0, 248412
GlobalTracer [candidate] (248.099 ms) : 0, 248099
AppSec [baseline] (27.553 ms) : 0, 27553
AppSec [candidate] (27.341 ms) : 0, 27341
Debugger [baseline] (6.241 ms) : 0, 6241
Debugger [candidate] (6.188 ms) : 0, 6188
Remote Config [baseline] (607.521 µs) : 0, 608
Remote Config [candidate] (587.276 µs) : 0, 587
Telemetry [baseline] (8.373 ms) : 0, 8373
Telemetry [candidate] (8.138 ms) : 0, 8138
IAST [baseline] (31.77 ms) : 0, 31770
IAST [candidate] (31.729 ms) : 0, 31729
section profiling
ProfilingAgent [baseline] (101.95 ms) : 0, 101950
ProfilingAgent [candidate] (101.567 ms) : 0, 101567
crashtracking [baseline] (1.437 ms) : 0, 1437
crashtracking [candidate] (1.442 ms) : 0, 1442
BytebuddyAgent [baseline] (719.179 ms) : 0, 719179
BytebuddyAgent [candidate] (717.716 ms) : 0, 717716
GlobalTracer [baseline] (235.604 ms) : 0, 235604
GlobalTracer [candidate] (235.038 ms) : 0, 235038
AppSec [baseline] (31.173 ms) : 0, 31173
AppSec [candidate] (31.137 ms) : 0, 31137
Debugger [baseline] (6.511 ms) : 0, 6511
Debugger [candidate] (6.464 ms) : 0, 6464
Remote Config [baseline] (721.403 µs) : 0, 721
Remote Config [candidate] (734.657 µs) : 0, 735
Telemetry [baseline] (16.746 ms) : 0, 16746
Telemetry [candidate] (16.684 ms) : 0, 16684
Profiling [baseline] (102.547 ms) : 0, 102547
Profiling [candidate] (102.157 ms) : 0, 102157
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.018 s) : 0, 1017855
Total [baseline] (8.679 s) : 0, 8678794
Agent [candidate] (1.018 s) : 0, 1017570
Total [candidate] (8.664 s) : 0, 8664239
section iast
Agent [baseline] (1.161 s) : 0, 1160771
Total [baseline] (9.292 s) : 0, 9292422
Agent [candidate] (1.15 s) : 0, 1150036
Total [candidate] (9.383 s) : 0, 9383035
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.018 s -
Agent iast 1.161 s 142.916 ms (14.0%)
Total tracing 8.679 s -
Total iast 9.292 s 613.628 ms (7.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.018 s -
Agent iast 1.15 s 132.466 ms (13.0%)
Total tracing 8.664 s -
Total iast 9.383 s 718.797 ms (8.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.46 ms) : 0, 1460
crashtracking [candidate] (1.438 ms) : 0, 1438
BytebuddyAgent [baseline] (685.389 ms) : 0, 685389
BytebuddyAgent [candidate] (685.82 ms) : 0, 685820
GlobalTracer [baseline] (257.218 ms) : 0, 257218
GlobalTracer [candidate] (257.472 ms) : 0, 257472
AppSec [baseline] (31.62 ms) : 0, 31620
AppSec [candidate] (31.557 ms) : 0, 31557
Debugger [baseline] (6.349 ms) : 0, 6349
Debugger [candidate] (6.312 ms) : 0, 6312
Remote Config [baseline] (686.209 µs) : 0, 686
Remote Config [candidate] (675.704 µs) : 0, 676
Telemetry [baseline] (14.219 ms) : 0, 14219
Telemetry [candidate] (13.369 ms) : 0, 13369
section iast
crashtracking [baseline] (1.482 ms) : 0, 1482
crashtracking [candidate] (1.453 ms) : 0, 1453
BytebuddyAgent [baseline] (814.013 ms) : 0, 814013
BytebuddyAgent [candidate] (806.626 ms) : 0, 806626
GlobalTracer [baseline] (249.404 ms) : 0, 249404
GlobalTracer [candidate] (246.892 ms) : 0, 246892
IAST [baseline] (32.152 ms) : 0, 32152
IAST [candidate] (30.839 ms) : 0, 30839
AppSec [baseline] (27.375 ms) : 0, 27375
AppSec [candidate] (28.262 ms) : 0, 28262
Debugger [baseline] (6.275 ms) : 0, 6275
Debugger [candidate] (6.165 ms) : 0, 6165
Remote Config [baseline] (607.354 µs) : 0, 607
Remote Config [candidate] (602.902 µs) : 0, 603
Telemetry [baseline] (8.398 ms) : 0, 8398
Telemetry [candidate] (8.296 ms) : 0, 8296
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master alexeyk/fixed-flaky-histogram-test
git_commit_date 1758571586 1758573786
git_commit_sha 7b1d89d acd784f
release_version 1.54.0-SNAPSHOT~7b1d89d384 1.54.0-SNAPSHOT~acd784fb1b
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1758575446 1758575446
ci_job_id 1142107323 1142107323
ci_pipeline_id 77223452 77223452
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-0rhk4nth 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-0rhk4nth 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 1 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:insecure-bank:tracing:high_load better
[-629.738µs; -392.427µs] or [-7.961%; -4.961%]		 unstable
[-38.427op/s; +118.302op/s] or [-6.559%; +20.193%]		 7.399ms 625.781op/s 7.910ms 585.844op/s
scenario:load:insecure-bank:profiling:high_load worse
[+242.656µs; +554.758µs] or [+2.809%; +6.421%]		 unstable
[-93.558op/s; +46.683op/s] or [-17.429%; +8.696%]		 9.038ms 513.375op/s 8.639ms 536.812op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load better
[-727.354µs; -321.251µs] or [-6.686%; -2.953%]		 unstable
[-31.562op/s; +74.500op/s] or [-7.385%; +17.431%]		 10.354ms 448.875op/s 10.878ms 427.406op/s
scenario:load:insecure-bank:iast:high_load better
[-660.944µs; -315.593µs] or [-6.686%; -3.193%]		 unstable
[-34.913op/s; +83.475op/s] or [-7.432%; +17.769%]		 9.397ms 494.062op/s 9.885ms 469.781op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.368 ms) : 4314, 4422
.   : milestone, 4368,
iast (9.885 ms) : 9719, 10052
.   : milestone, 9885,
iast_FULL (14.202 ms) : 13916, 14487
.   : milestone, 14202,
iast_GLOBAL (10.878 ms) : 10684, 11072
.   : milestone, 10878,
profiling (8.639 ms) : 8509, 8769
.   : milestone, 8639,
tracing (7.91 ms) : 7792, 8028
.   : milestone, 7910,
section candidate
no_agent (4.326 ms) : 4279, 4373
.   : milestone, 4326,
iast (9.397 ms) : 9243, 9551
.   : milestone, 9397,
iast_FULL (14.465 ms) : 14172, 14758
.   : milestone, 14465,
iast_GLOBAL (10.354 ms) : 10171, 10537
.   : milestone, 10354,
profiling (9.038 ms) : 8879, 9196
.   : milestone, 9038,
tracing (7.399 ms) : 7297, 7501
.   : milestone, 7399,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.368 ms [4.314 ms, 4.422 ms] -
iast 9.885 ms [9.719 ms, 10.052 ms] 5.517 ms (126.3%)
iast_FULL 14.202 ms [13.916 ms, 14.487 ms] 9.833 ms (225.1%)
iast_GLOBAL 10.878 ms [10.684 ms, 11.072 ms] 6.51 ms (149.0%)
profiling 8.639 ms [8.509 ms, 8.769 ms] 4.271 ms (97.8%)
tracing 7.91 ms [7.792 ms, 8.028 ms] 3.542 ms (81.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.326 ms [4.279 ms, 4.373 ms] -
iast 9.397 ms [9.243 ms, 9.551 ms] 5.071 ms (117.2%)
iast_FULL 14.465 ms [14.172 ms, 14.758 ms] 10.139 ms (234.4%)
iast_GLOBAL 10.354 ms [10.171 ms, 10.537 ms] 6.028 ms (139.3%)
profiling 9.038 ms [8.879 ms, 9.196 ms] 4.712 ms (108.9%)
tracing 7.399 ms [7.297 ms, 7.501 ms] 3.073 ms (71.0%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384
    dateFormat X
    axisFormat %s
section baseline
no_agent (36.681 ms) : 36394, 36968
.   : milestone, 36681,
appsec (48.546 ms) : 48115, 48976
.   : milestone, 48546,
code_origins (43.095 ms) : 42717, 43473
.   : milestone, 43095,
iast (45.259 ms) : 44860, 45659
.   : milestone, 45259,
profiling (48.371 ms) : 47931, 48812
.   : milestone, 48371,
tracing (42.948 ms) : 42579, 43317
.   : milestone, 42948,
section candidate
no_agent (36.249 ms) : 35960, 36539
.   : milestone, 36249,
appsec (48.147 ms) : 47724, 48571
.   : milestone, 48147,
code_origins (43.784 ms) : 43407, 44161
.   : milestone, 43784,
iast (44.89 ms) : 44487, 45293
.   : milestone, 44890,
profiling (48.233 ms) : 47805, 48662
.   : milestone, 48233,
tracing (43.933 ms) : 43559, 44307
.   : milestone, 43933,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 36.681 ms [36.394 ms, 36.968 ms] -
appsec 48.546 ms [48.115 ms, 48.976 ms] 11.864 ms (32.3%)
code_origins 43.095 ms [42.717 ms, 43.473 ms] 6.414 ms (17.5%)
iast 45.259 ms [44.86 ms, 45.659 ms] 8.578 ms (23.4%)
profiling 48.371 ms [47.931 ms, 48.812 ms] 11.69 ms (31.9%)
tracing 42.948 ms [42.579 ms, 43.317 ms] 6.266 ms (17.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 36.249 ms [35.96 ms, 36.539 ms] -
appsec 48.147 ms [47.724 ms, 48.571 ms] 11.898 ms (32.8%)
code_origins 43.784 ms [43.407 ms, 44.161 ms] 7.535 ms (20.8%)
iast 44.89 ms [44.487 ms, 45.293 ms] 8.641 ms (23.8%)
profiling 48.233 ms [47.805 ms, 48.662 ms] 11.984 ms (33.1%)
tracing 43.933 ms [43.559 ms, 44.307 ms] 7.683 ms (21.2%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master alexeyk/fixed-flaky-histogram-test
git_commit_date 1758571586 1758573786
git_commit_sha 7b1d89d acd784f
release_version 1.54.0-SNAPSHOT~7b1d89d384 1.54.0-SNAPSHOT~acd784fb1b
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1758575967 1758575967
ci_job_id 1142107324 1142107324
ci_pipeline_id 77223452 77223452
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-2-hgmey5yu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-2-hgmey5yu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.471 ms) : 1460, 1483
.   : milestone, 1471,
appsec (3.713 ms) : 3497, 3930
.   : milestone, 3713,
iast (2.193 ms) : 2131, 2256
.   : milestone, 2193,
iast_GLOBAL (2.243 ms) : 2180, 2306
.   : milestone, 2243,
profiling (2.039 ms) : 1989, 2090
.   : milestone, 2039,
tracing (2.026 ms) : 1976, 2076
.   : milestone, 2026,
section candidate
no_agent (1.469 ms) : 1457, 1480
.   : milestone, 1469,
appsec (3.621 ms) : 3408, 3835
.   : milestone, 3621,
iast (2.202 ms) : 2139, 2264
.   : milestone, 2202,
iast_GLOBAL (2.239 ms) : 2176, 2302
.   : milestone, 2239,
profiling (2.033 ms) : 1982, 2083
.   : milestone, 2033,
tracing (2.02 ms) : 1970, 2069
.   : milestone, 2020,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.471 ms [1.46 ms, 1.483 ms] -
appsec 3.713 ms [3.497 ms, 3.93 ms] 2.242 ms (152.4%)
iast 2.193 ms [2.131 ms, 2.256 ms] 722.058 µs (49.1%)
iast_GLOBAL 2.243 ms [2.18 ms, 2.306 ms] 771.723 µs (52.5%)
profiling 2.039 ms [1.989 ms, 2.09 ms] 568.086 µs (38.6%)
tracing 2.026 ms [1.976 ms, 2.076 ms] 554.607 µs (37.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.469 ms [1.457 ms, 1.48 ms] -
appsec 3.621 ms [3.408 ms, 3.835 ms] 2.153 ms (146.6%)
iast 2.202 ms [2.139 ms, 2.264 ms] 733.087 µs (49.9%)
iast_GLOBAL 2.239 ms [2.176 ms, 2.302 ms] 770.431 µs (52.5%)
profiling 2.033 ms [1.982 ms, 2.083 ms] 564.059 µs (38.4%)
tracing 2.02 ms [1.97 ms, 2.069 ms] 551.048 µs (37.5%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~acd784fb1b, baseline=1.54.0-SNAPSHOT~7b1d89d384
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.068 s) : 15068000, 15068000
.   : milestone, 15068000,
appsec (15.037 s) : 15037000, 15037000
.   : milestone, 15037000,
iast (18.56 s) : 18560000, 18560000
.   : milestone, 18560000,
iast_GLOBAL (18.135 s) : 18135000, 18135000
.   : milestone, 18135000,
profiling (15.633 s) : 15633000, 15633000
.   : milestone, 15633000,
tracing (15.071 s) : 15071000, 15071000
.   : milestone, 15071000,
section candidate
no_agent (15.228 s) : 15228000, 15228000
.   : milestone, 15228000,
appsec (15.051 s) : 15051000, 15051000
.   : milestone, 15051000,
iast (18.362 s) : 18362000, 18362000
.   : milestone, 18362000,
iast_GLOBAL (18.001 s) : 18001000, 18001000
.   : milestone, 18001000,
profiling (15.352 s) : 15352000, 15352000
.   : milestone, 15352000,
tracing (15.117 s) : 15117000, 15117000
.   : milestone, 15117000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.068 s [15.068 s, 15.068 s] -
appsec 15.037 s [15.037 s, 15.037 s] -31.0 ms (-0.2%)
iast 18.56 s [18.56 s, 18.56 s] 3.492 s (23.2%)
iast_GLOBAL 18.135 s [18.135 s, 18.135 s] 3.067 s (20.4%)
profiling 15.633 s [15.633 s, 15.633 s] 565.0 ms (3.7%)
tracing 15.071 s [15.071 s, 15.071 s] 3.0 ms (0.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.228 s [15.228 s, 15.228 s] -
appsec 15.051 s [15.051 s, 15.051 s] -177.0 ms (-1.2%)
iast 18.362 s [18.362 s, 18.362 s] 3.134 s (20.6%)
iast_GLOBAL 18.001 s [18.001 s, 18.001 s] 2.773 s (18.2%)
profiling 15.352 s [15.352 s, 15.352 s] 124.0 ms (0.8%)
tracing 15.117 s [15.117 s, 15.117 s] -111.0 ms (-0.7%)

bric3
bric3 reviewed Sep 23, 2025
View reviewed changes
Copy link
Contributor

@bric3 bric3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of questions:

  1. Alternatively would it be better to retry ?
  2. I haven't looked deeply into the test, but I noticed increasing relativeAccuracy numbers, is there a pattern to be careful to follow ?

Otherwise, that looks OK to me.

@AlexeyKuznetsov-DD
Copy link
Contributor Author

AlexeyKuznetsov-DD commented Sep 23, 2025

Couple of questions:

  1. Alternatively would it be better to retry ?
  2. I haven't looked deeply into the test, but I noticed increasing relativeAccuracy numbers, is there a pattern to be careful to follow ?

Otherwise, that looks OK to me.

  1. The test already has retries at the Gradle level. However, this makes it show up as flaky in the CI-Vis dashboard, which is why adding another retry doesn’t really improve things, just make test code a bit complex.
  2. This particular test seems inconsistent to me. Originally it was declared like this:
    normal     |   10000  | [1000D, 10D]    | 0.01
    normal     |   100000 | [1000D, 10D]    | 0.02

The odd part is that the second test uses a larger sample size (100k vs 10k) with otherwise identical parameters, yet its relativeAccuracy is higher. Statistically, larger samples should give more stable results, not less. So having a relativeAccuracy for the smaller sample makes little sense.

@bric3
Copy link
Contributor

bric3 commented Sep 23, 2025

Yeah that seemed odd to me for the same reasons you stated, so let's go ahead with this change :)

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD merged commit 11e677e into master Sep 23, 2025
551 checks passed
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD deleted the alexeyk/fixed-flaky-histogram-test branch September 23, 2025 20:06
@github-actions github-actions bot added this to the 1.54.0 milestone Sep 23, 2025
mhlidd pushed a commit that referenced this pull request Sep 24, 2025
@AlexeyKuznetsov-DD @mhlidd
Raised relativeAccuracy to 0.2 since 0.1 causes ~16% random f…
25e9109 
…ailures at `n=10,000` due to expected fluctuations. (#9588)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bric3 bric3 bric3 approved these changes

@sarahchen6 sarahchen6 sarahchen6 approved these changes

@mcculls mcculls Awaiting requested review from mcculls

@PerfectSlayer PerfectSlayer Awaiting requested review from PerfectSlayer

Assignees

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD

Labels

comp: testing Testing tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.54.0

Development

Successfully merging this pull request may close these issues.

4 participants

@AlexeyKuznetsov-DD @bric3 @sarahchen6