Remove non-essential IAST benchmark scenarios #8999

smola · 2025-06-17T06:49:16Z

What Does This Do

Remove some IAST benchmark scenarios of unusual setups that we're currently not optimizing for: iast_INACTIVE, iast_TELEMETRY_OFF, iast_HARDCODED_SECRET_DISABLED.

Motivation

Make room for additional iterations for used scenarios.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-06-17T07:25:49Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/remove-benchmark-scenarios
git_commit_date	1750142716	1750143638
git_commit_sha	`3a93c7c`	7b7bea6a83
release_version	1.50.0-SNAPSHOT~3a93c7c839	1.50.0-SNAPSHOT~07b7bea6a83

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1750144894	1750144894
ci_job_id	985064956	985064956
ci_pipeline_id	67942196	67942196
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-2ousnmcz-project-304-concurrent-2-ndi5rwga 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-2ousnmcz-project-304-concurrent-2-ndi5rwga 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 42 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~07b7bea6a83, baseline=1.50.0-SNAPSHOT~3a93c7c839

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.041 s) : 0, 1041105
Total [baseline] (10.737 s) : 0, 10736663
Agent [candidate] (1.031 s) : 0, 1030672
Total [candidate] (10.786 s) : 0, 10785857
section appsec
Agent [baseline] (1.188 s) : 0, 1188073
Total [baseline] (10.85 s) : 0, 10849631
Agent [candidate] (1.182 s) : 0, 1181690
Total [candidate] (10.787 s) : 0, 10787162
section iast
Agent [baseline] (1.164 s) : 0, 1163633
Total [baseline] (10.863 s) : 0, 10863272
Agent [candidate] (1.173 s) : 0, 1173222
Total [candidate] (10.942 s) : 0, 10941716
section profiling
Agent [baseline] (1.276 s) : 0, 1275646
Total [baseline] (10.935 s) : 0, 10935071
Agent [candidate] (1.281 s) : 0, 1281074
Total [candidate] (11.012 s) : 0, 11012187

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.041 s	-
Agent	appsec	1.188 s	146.969 ms (14.1%)
Agent	iast	1.164 s	122.528 ms (11.8%)
Agent	profiling	1.276 s	234.541 ms (22.5%)
Total	tracing	10.737 s	-
Total	appsec	10.85 s	112.968 ms (1.1%)
Total	iast	10.863 s	126.609 ms (1.2%)
Total	profiling	10.935 s	198.408 ms (1.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.031 s	-
Agent	appsec	1.182 s	151.018 ms (14.7%)
Agent	iast	1.173 s	142.549 ms (13.8%)
Agent	profiling	1.281 s	250.401 ms (24.3%)
Total	tracing	10.786 s	-
Total	appsec	10.787 s	1.305 ms (0.0%)
Total	iast	10.942 s	155.859 ms (1.4%)
Total	profiling	11.012 s	226.33 ms (2.1%)

gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~07b7bea6a83, baseline=1.50.0-SNAPSHOT~3a93c7c839

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (693.814 ms) : 0, 693814
BytebuddyAgent [candidate] (686.094 ms) : 0, 686094
GlobalTracer [baseline] (243.921 ms) : 0, 243921
GlobalTracer [candidate] (241.913 ms) : 0, 241913
AppSec [baseline] (62.755 ms) : 0, 62755
AppSec [candidate] (59.406 ms) : 0, 59406
Debugger [baseline] (6.116 ms) : 0, 6116
Debugger [candidate] (6.917 ms) : 0, 6917
Remote Config [baseline] (738.366 µs) : 0, 738
Remote Config [candidate] (764.559 µs) : 0, 765
Telemetry [baseline] (9.942 ms) : 0, 9942
Telemetry [candidate] (12.042 ms) : 0, 12042
section appsec
BytebuddyAgent [baseline] (712.021 ms) : 0, 712021
BytebuddyAgent [candidate] (707.856 ms) : 0, 707856
GlobalTracer [baseline] (236.593 ms) : 0, 236593
GlobalTracer [candidate] (235.614 ms) : 0, 235614
IAST [baseline] (22.035 ms) : 0, 22035
IAST [candidate] (21.793 ms) : 0, 21793
AppSec [baseline] (179.808 ms) : 0, 179808
AppSec [candidate] (179.132 ms) : 0, 179132
Debugger [baseline] (5.935 ms) : 0, 5935
Debugger [candidate] (5.846 ms) : 0, 5846
Remote Config [baseline] (611.098 µs) : 0, 611
Remote Config [candidate] (600.032 µs) : 0, 600
Telemetry [baseline] (7.356 ms) : 0, 7356
Telemetry [candidate] (7.28 ms) : 0, 7280
section iast
BytebuddyAgent [baseline] (810.697 ms) : 0, 810697
BytebuddyAgent [candidate] (817.113 ms) : 0, 817113
GlobalTracer [baseline] (232.563 ms) : 0, 232563
GlobalTracer [candidate] (234.251 ms) : 0, 234251
IAST [baseline] (25.51 ms) : 0, 25510
IAST [candidate] (25.568 ms) : 0, 25568
AppSec [baseline] (56.588 ms) : 0, 56588
AppSec [candidate] (57.585 ms) : 0, 57585
Debugger [baseline] (6.056 ms) : 0, 6056
Debugger [candidate] (6.133 ms) : 0, 6133
Remote Config [baseline] (588.145 µs) : 0, 588
Remote Config [candidate] (611.581 µs) : 0, 612
Telemetry [baseline] (7.931 ms) : 0, 7931
Telemetry [candidate] (8.079 ms) : 0, 8079
section profiling
BytebuddyAgent [baseline] (676.039 ms) : 0, 676039
BytebuddyAgent [candidate] (678.2 ms) : 0, 678200
GlobalTracer [baseline] (360.838 ms) : 0, 360838
GlobalTracer [candidate] (363.085 ms) : 0, 363085
AppSec [baseline] (64.756 ms) : 0, 64756
AppSec [candidate] (65.239 ms) : 0, 65239
Debugger [baseline] (6.162 ms) : 0, 6162
Debugger [candidate] (6.254 ms) : 0, 6254
Remote Config [baseline] (654.032 µs) : 0, 654
Remote Config [candidate] (647.848 µs) : 0, 648
Telemetry [baseline] (8.251 ms) : 0, 8251
Telemetry [candidate] (8.252 ms) : 0, 8252
ProfilingAgent [baseline] (108.145 ms) : 0, 108145
ProfilingAgent [candidate] (108.396 ms) : 0, 108396
Profiling [baseline] (108.171 ms) : 0, 108171
Profiling [candidate] (108.42 ms) : 0, 108420

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~07b7bea6a83, baseline=1.50.0-SNAPSHOT~3a93c7c839

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.036 s) : 0, 1035547
Total [baseline] (8.615 s) : 0, 8614578
Agent [candidate] (1.043 s) : 0, 1043205
Total [candidate] (8.579 s) : 0, 8579416
section iast
Agent [baseline] (1.156 s) : 0, 1156317
Total [baseline] (9.218 s) : 0, 9217610
Agent [candidate] (1.157 s) : 0, 1156779
Total [candidate] (9.259 s) : 0, 9258610

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.036 s	-
Agent	iast	1.156 s	120.771 ms (11.7%)
Total	tracing	8.615 s	-
Total	iast	9.218 s	603.032 ms (7.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	iast	1.157 s	113.574 ms (10.9%)
Total	tracing	8.579 s	-
Total	iast	9.259 s	679.194 ms (7.9%)

gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~07b7bea6a83, baseline=1.50.0-SNAPSHOT~3a93c7c839

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (690.148 ms) : 0, 690148
BytebuddyAgent [candidate] (694.511 ms) : 0, 694511
GlobalTracer [baseline] (242.636 ms) : 0, 242636
GlobalTracer [candidate] (243.926 ms) : 0, 243926
AppSec [baseline] (62.311 ms) : 0, 62311
AppSec [candidate] (61.248 ms) : 0, 61248
Debugger [baseline] (6.137 ms) : 0, 6137
Debugger [candidate] (6.181 ms) : 0, 6181
Remote Config [baseline] (731.189 µs) : 0, 731
Remote Config [candidate] (751.912 µs) : 0, 752
Telemetry [baseline] (9.829 ms) : 0, 9829
Telemetry [candidate] (12.803 ms) : 0, 12803
section iast
BytebuddyAgent [baseline] (805.068 ms) : 0, 805068
BytebuddyAgent [candidate] (805.166 ms) : 0, 805166
GlobalTracer [baseline] (231.376 ms) : 0, 231376
GlobalTracer [candidate] (231.469 ms) : 0, 231469
AppSec [baseline] (54.868 ms) : 0, 54868
AppSec [candidate] (54.146 ms) : 0, 54146
Debugger [baseline] (5.999 ms) : 0, 5999
Debugger [candidate] (6.078 ms) : 0, 6078
Remote Config [baseline] (583.58 µs) : 0, 584
Remote Config [candidate] (602.891 µs) : 0, 603
Telemetry [baseline] (7.944 ms) : 0, 7944
Telemetry [candidate] (8.034 ms) : 0, 8034
IAST [baseline] (26.973 ms) : 0, 26973
IAST [candidate] (27.775 ms) : 0, 27775

Load

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/remove-benchmark-scenarios
git_commit_date	1750142716	1750143644
git_commit_sha	`3a93c7c`	7b7bea6a83
release_version	1.50.0-SNAPSHOT~3a93c7c839	1.50.0-SNAPSHOT~07b7bea6a83

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1750145110	1750145110
ci_job_id	985064958	985064958
ci_pipeline_id	67942196	67942196
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-2ousnmcz-project-304-concurrent-3-vo1lyyon 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-2ousnmcz-project-304-concurrent-3-vo1lyyon 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~07b7bea6a83, baseline=1.50.0-SNAPSHOT~3a93c7c839
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (2.402 ms) : 2354, 2450
.   : milestone, 2402,
iast (2.195 ms) : 2134, 2256
.   : milestone, 2195,
iast_GLOBAL (2.228 ms) : 2167, 2289
.   : milestone, 2228,
profiling (2.471 ms) : 2299, 2643
.   : milestone, 2471,
tracing (2.011 ms) : 1964, 2058
.   : milestone, 2011,
section candidate
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (2.395 ms) : 2347, 2443
.   : milestone, 2395,
iast (2.191 ms) : 2130, 2251
.   : milestone, 2191,
iast_GLOBAL (2.224 ms) : 2163, 2285
.   : milestone, 2224,
profiling (2.027 ms) : 1978, 2077
.   : milestone, 2027,
tracing (2.002 ms) : 1955, 2049
.   : milestone, 2002,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.463 ms, 1.486 ms]	-
appsec	2.402 ms [2.354 ms, 2.45 ms]	927.363 µs (62.9%)
iast	2.195 ms [2.134 ms, 2.256 ms]	720.61 µs (48.9%)
iast_GLOBAL	2.228 ms [2.167 ms, 2.289 ms]	753.412 µs (51.1%)
profiling	2.471 ms [2.299 ms, 2.643 ms]	996.551 µs (67.6%)
tracing	2.011 ms [1.964 ms, 2.058 ms]	536.328 µs (36.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.463 ms, 1.486 ms]	-
appsec	2.395 ms [2.347 ms, 2.443 ms]	920.59 µs (62.4%)
iast	2.191 ms [2.13 ms, 2.251 ms]	716.511 µs (48.6%)
iast_GLOBAL	2.224 ms [2.163 ms, 2.285 ms]	750.253 µs (50.9%)
profiling	2.027 ms [1.978 ms, 2.077 ms]	553.15 µs (37.5%)
tracing	2.002 ms [1.955 ms, 2.049 ms]	528.025 µs (35.8%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~07b7bea6a83, baseline=1.50.0-SNAPSHOT~3a93c7c839
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.421 s) : 15421000, 15421000
.   : milestone, 15421000,
appsec (14.836 s) : 14836000, 14836000
.   : milestone, 14836000,
iast (18.788 s) : 18788000, 18788000
.   : milestone, 18788000,
iast_GLOBAL (18.181 s) : 18181000, 18181000
.   : milestone, 18181000,
profiling (15.327 s) : 15327000, 15327000
.   : milestone, 15327000,
tracing (14.969 s) : 14969000, 14969000
.   : milestone, 14969000,
section candidate
no_agent (15.409 s) : 15409000, 15409000
.   : milestone, 15409000,
appsec (14.913 s) : 14913000, 14913000
.   : milestone, 14913000,
iast (19.117 s) : 19117000, 19117000
.   : milestone, 19117000,
iast_GLOBAL (17.757 s) : 17757000, 17757000
.   : milestone, 17757000,
profiling (15.35 s) : 15350000, 15350000
.   : milestone, 15350000,
tracing (14.832 s) : 14832000, 14832000
.   : milestone, 14832000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.421 s [15.421 s, 15.421 s]	-
appsec	14.836 s [14.836 s, 14.836 s]	-585.0 ms (-3.8%)
iast	18.788 s [18.788 s, 18.788 s]	3.367 s (21.8%)
iast_GLOBAL	18.181 s [18.181 s, 18.181 s]	2.76 s (17.9%)
profiling	15.327 s [15.327 s, 15.327 s]	-94.0 ms (-0.6%)
tracing	14.969 s [14.969 s, 14.969 s]	-452.0 ms (-2.9%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.409 s [15.409 s, 15.409 s]	-
appsec	14.913 s [14.913 s, 14.913 s]	-496.0 ms (-3.2%)
iast	19.117 s [19.117 s, 19.117 s]	3.708 s (24.1%)
iast_GLOBAL	17.757 s [17.757 s, 17.757 s]	2.348 s (15.2%)
profiling	15.35 s [15.35 s, 15.35 s]	-59.0 ms (-0.4%)
tracing	14.832 s [14.832 s, 14.832 s]	-577.0 ms (-3.7%)

Remove non-essential IAST benchmark scenarios

07b7bea

smola requested a review from a team as a code owner June 17, 2025 06:49

smola requested review from sarahchen6 and removed request for a team June 17, 2025 06:49

smola added tag: no release notes Changes to exclude from release notes comp: asm iast Application Security Management (IAST) labels Jun 17, 2025

smola requested a review from manuel-alvarez-alvarez June 17, 2025 06:50

manuel-alvarez-alvarez approved these changes Jun 17, 2025

View reviewed changes

sarahchen6 approved these changes Jun 17, 2025

View reviewed changes

smola added tag: no release notes Changes to exclude from release notes and removed tag: no release notes Changes to exclude from release notes labels Jun 17, 2025

smola merged commit a4149e2 into master Jun 17, 2025
316 of 317 checks passed

smola deleted the smola/remove-benchmark-scenarios branch June 17, 2025 13:26

github-actions bot added this to the 1.50.0 milestone Jun 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Remove non-essential IAST benchmark scenarios #8999

Remove non-essential IAST benchmark scenarios #8999

Uh oh!

smola commented Jun 17, 2025

Uh oh!

pr-commenter bot commented Jun 17, 2025

Uh oh!

Uh oh!

Uh oh!

Remove non-essential IAST benchmark scenarios #8999

Remove non-essential IAST benchmark scenarios #8999

Uh oh!

Conversation

smola commented Jun 17, 2025

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Jun 17, 2025

Benchmarks

Startup

Parameters

Summary

Load

Dacapo

Parameters

Summary

Uh oh!

Uh oh!

Uh oh!