Skip to content

Implement Selenium support in CI Visibility #6799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Apr 4, 2024
Merged

Implement Selenium support in CI Visibility #6799

merged 8 commits into from
Apr 4, 2024

Conversation

@nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Mar 11, 2024 •
edited
Loading

What Does This Do

Implements support for Selenium framework integration in CI Visibility.
Tests that are using Selenium bindings for Java are tagger appropriately.
If the tests are using Datadog RUM, Selenium is used to provide test trace context to RUM.

Additional notes:

Here are corresponding changes in RUM

Jira ticket: CIVIS-9354

@nikita-tkachenko-datadog nikita-tkachenko-datadog added comp: ci visibility Continuous Integration Visibility run-tests: all Run all tests labels Mar 11, 2024
@pr-commenter
Copy link

pr-commenter bot commented Mar 11, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/selenium-support
git_commit_date 1710755773 1710766564
git_commit_sha b708147 bcc535c
release_version 1.32.0-SNAPSHOT~b708147d74 1.32.0-SNAPSHOT~bcc535c3b1
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1710769795 1710769795
ci_job_id 462123456 462123456
ci_pipeline_id 30286169 30286169
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 2 performance improvements and 2 performance regressions! Performance is the same for 45 metrics, 14 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:tracing:Remote Config better
[-394.622µs; -348.615µs] or [-35.482%; -31.345%]		 0.741ms 1.112ms
scenario:startup:insecure-bank:tracing:Telemetry worse
[+275.442µs; +542.196µs] or [+3.738%; +7.358%]		 7.777ms 7.368ms
scenario:startup:petclinic:iast:Remote Config worse
[+13.113µs; +49.776µs] or [+2.301%; +8.734%]		 601.361µs 569.917µs
scenario:startup:petclinic:tracing:Remote Config better
[-358.898µs; -300.461µs] or [-33.308%; -27.885%]		 0.748ms 1.078ms
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~bcc535c3b1, baseline=1.32.0-SNAPSHOT~b708147d74

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.079 s) : 0, 1078687
Total [baseline] (8.558 s) : 0, 8558309
Agent [candidate] (1.082 s) : 0, 1082026
Total [candidate] (8.559 s) : 0, 8559098
section iast
Agent [baseline] (1.222 s) : 0, 1221770
Total [baseline] (9.048 s) : 0, 9047566
Agent [candidate] (1.209 s) : 0, 1209412
Total [candidate] (9.08 s) : 0, 9080080
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.208 s) : 0, 1208228
Total [baseline] (9.034 s) : 0, 9034243
Agent [candidate] (1.209 s) : 0, 1209484
Total [candidate] (9.047 s) : 0, 9046809
section iast_TELEMETRY_OFF
Agent [baseline] (1.205 s) : 0, 1205199
Total [baseline] (9.036 s) : 0, 9035574
Agent [candidate] (1.207 s) : 0, 1206941
Total [candidate] (9.088 s) : 0, 9087819
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.079 s -
Agent iast 1.222 s 143.083 ms (13.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.208 s 129.541 ms (12.0%)
Agent iast_TELEMETRY_OFF 1.205 s 126.511 ms (11.7%)
Total tracing 8.558 s -
Total iast 9.048 s 489.257 ms (5.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.034 s 475.934 ms (5.6%)
Total iast_TELEMETRY_OFF 9.036 s 477.265 ms (5.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.082 s -
Agent iast 1.209 s 127.385 ms (11.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.209 s 127.458 ms (11.8%)
Agent iast_TELEMETRY_OFF 1.207 s 124.915 ms (11.5%)
Total tracing 8.559 s -
Total iast 9.08 s 520.983 ms (6.1%)
Total iast_HARDCODED_SECRET_DISABLED 9.047 s 487.711 ms (5.7%)
Total iast_TELEMETRY_OFF 9.088 s 528.721 ms (6.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~bcc535c3b1, baseline=1.32.0-SNAPSHOT~b708147d74

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (695.937 ms) : 0, 695937
BytebuddyAgent [candidate] (697.488 ms) : 0, 697488
GlobalTracer [baseline] (291.13 ms) : 0, 291130
GlobalTracer [candidate] (292.883 ms) : 0, 292883
AppSec [baseline] (48.868 ms) : 0, 48868
AppSec [candidate] (48.867 ms) : 0, 48867
Remote Config [baseline] (1.112 ms) : 0, 1112
Remote Config [candidate] (740.571 µs) : 0, 741
Telemetry [baseline] (7.368 ms) : 0, 7368
Telemetry [candidate] (7.777 ms) : 0, 7777
section iast
BytebuddyAgent [baseline] (812.376 ms) : 0, 812376
BytebuddyAgent [candidate] (803.037 ms) : 0, 803037
GlobalTracer [baseline] (292.05 ms) : 0, 292050
GlobalTracer [candidate] (290.19 ms) : 0, 290190
AppSec [baseline] (50.903 ms) : 0, 50903
AppSec [candidate] (49.708 ms) : 0, 49708
IAST [baseline] (22.612 ms) : 0, 22612
IAST [candidate] (24.78 ms) : 0, 24780
Remote Config [baseline] (584.636 µs) : 0, 585
Remote Config [candidate] (618.644 µs) : 0, 619
Telemetry [baseline] (8.237 ms) : 0, 8237
Telemetry [candidate] (6.614 ms) : 0, 6614
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.486 ms) : 0, 802486
BytebuddyAgent [candidate] (803.187 ms) : 0, 803187
GlobalTracer [baseline] (289.45 ms) : 0, 289450
GlobalTracer [candidate] (290.567 ms) : 0, 290567
AppSec [baseline] (49.835 ms) : 0, 49835
AppSec [candidate] (51.617 ms) : 0, 51617
IAST [baseline] (24.092 ms) : 0, 24092
IAST [candidate] (22.334 ms) : 0, 22334
Remote Config [baseline] (573.792 µs) : 0, 574
Remote Config [candidate] (606.332 µs) : 0, 606
Telemetry [baseline] (7.334 ms) : 0, 7334
Telemetry [candidate] (6.708 ms) : 0, 6708
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (798.654 ms) : 0, 798654
BytebuddyAgent [candidate] (799.38 ms) : 0, 799380
GlobalTracer [baseline] (290.717 ms) : 0, 290717
GlobalTracer [candidate] (291.648 ms) : 0, 291648
AppSec [baseline] (49.099 ms) : 0, 49099
AppSec [candidate] (47.329 ms) : 0, 47329
IAST [baseline] (24.97 ms) : 0, 24970
IAST [candidate] (26.274 ms) : 0, 26274
Remote Config [baseline] (576.114 µs) : 0, 576
Remote Config [candidate] (566.921 µs) : 0, 567
Telemetry [baseline] (6.56 ms) : 0, 6560
Telemetry [candidate] (7.165 ms) : 0, 7165
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~bcc535c3b1, baseline=1.32.0-SNAPSHOT~b708147d74

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.078 s) : 0, 1077521
Total [baseline] (9.202 s) : 0, 9202081
Agent [candidate] (1.084 s) : 0, 1083717
Total [candidate] (9.183 s) : 0, 9182600
section appsec
Agent [baseline] (1.202 s) : 0, 1201705
Total [baseline] (9.286 s) : 0, 9285773
Agent [candidate] (1.211 s) : 0, 1210889
Total [candidate] (9.317 s) : 0, 9316825
section iast
Agent [baseline] (1.205 s) : 0, 1205149
Total [baseline] (9.349 s) : 0, 9348780
Agent [candidate] (1.217 s) : 0, 1217383
Total [candidate] (9.401 s) : 0, 9400995
section profiling
Agent [baseline] (1.274 s) : 0, 1274067
Total [baseline] (9.383 s) : 0, 9382688
Agent [candidate] (1.283 s) : 0, 1283288
Total [candidate] (9.389 s) : 0, 9388721
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.078 s -
Agent appsec 1.202 s 124.184 ms (11.5%)
Agent iast 1.205 s 127.628 ms (11.8%)
Agent profiling 1.274 s 196.545 ms (18.2%)
Total tracing 9.202 s -
Total appsec 9.286 s 83.691 ms (0.9%)
Total iast 9.349 s 146.699 ms (1.6%)
Total profiling 9.383 s 180.607 ms (2.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.084 s -
Agent appsec 1.211 s 127.172 ms (11.7%)
Agent iast 1.217 s 133.666 ms (12.3%)
Agent profiling 1.283 s 199.571 ms (18.4%)
Total tracing 9.183 s -
Total appsec 9.317 s 134.225 ms (1.5%)
Total iast 9.401 s 218.395 ms (2.4%)
Total profiling 9.389 s 206.122 ms (2.2%) 
gantt
    title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~bcc535c3b1, baseline=1.32.0-SNAPSHOT~b708147d74

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (694.708 ms) : 0, 694708
BytebuddyAgent [candidate] (698.831 ms) : 0, 698831
GlobalTracer [baseline] (290.814 ms) : 0, 290814
GlobalTracer [candidate] (292.86 ms) : 0, 292860
AppSec [baseline] (49.162 ms) : 0, 49162
AppSec [candidate] (49.015 ms) : 0, 49015
Remote Config [baseline] (1.078 ms) : 0, 1078
Remote Config [candidate] (747.822 µs) : 0, 748
Telemetry [baseline] (7.516 ms) : 0, 7516
Telemetry [candidate] (7.807 ms) : 0, 7807
section appsec
BytebuddyAgent [baseline] (697.087 ms) : 0, 697087
BytebuddyAgent [candidate] (702.718 ms) : 0, 702718
GlobalTracer [baseline] (291.428 ms) : 0, 291428
GlobalTracer [candidate] (294.356 ms) : 0, 294356
AppSec [baseline] (153.467 ms) : 0, 153467
AppSec [candidate] (153.759 ms) : 0, 153759
IAST [baseline] (17.833 ms) : 0, 17833
IAST [candidate] (17.936 ms) : 0, 17936
Remote Config [baseline] (614.566 µs) : 0, 615
Remote Config [candidate] (605.473 µs) : 0, 605
Telemetry [baseline] (6.868 ms) : 0, 6868
Telemetry [candidate] (6.828 ms) : 0, 6828
section iast
BytebuddyAgent [baseline] (800.909 ms) : 0, 800909
BytebuddyAgent [candidate] (809.878 ms) : 0, 809878
GlobalTracer [baseline] (288.593 ms) : 0, 288593
GlobalTracer [candidate] (291.462 ms) : 0, 291462
AppSec [baseline] (48.812 ms) : 0, 48812
AppSec [candidate] (51.775 ms) : 0, 51775
IAST [baseline] (25.339 ms) : 0, 25339
IAST [candidate] (21.585 ms) : 0, 21585
Remote Config [baseline] (569.917 µs) : 0, 570
Remote Config [candidate] (601.361 µs) : 0, 601
Telemetry [baseline] (6.593 ms) : 0, 6593
Telemetry [candidate] (7.445 ms) : 0, 7445
section profiling
BytebuddyAgent [baseline] (689.544 ms) : 0, 689544
BytebuddyAgent [candidate] (694.236 ms) : 0, 694236
GlobalTracer [baseline] (375.521 ms) : 0, 375521
GlobalTracer [candidate] (378.917 ms) : 0, 378917
AppSec [baseline] (49.931 ms) : 0, 49931
AppSec [candidate] (50.003 ms) : 0, 50003
Remote Config [baseline] (928.881 µs) : 0, 929
Remote Config [candidate] (799.829 µs) : 0, 800
Telemetry [baseline] (7.285 ms) : 0, 7285
Telemetry [candidate] (7.508 ms) : 0, 7508
ProfilingAgent [baseline] (94.491 ms) : 0, 94491
ProfilingAgent [candidate] (95.213 ms) : 0, 95213
Profiling [baseline] (94.515 ms) : 0, 94515
Profiling [candidate] (95.236 ms) : 0, 95236
Loading

Load

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~bcc535c3b1, baseline=1.32.0-SNAPSHOT~b708147d74
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.57 µs) : 347, 388
.   : milestone, 368,
iast (471.259 µs) : 450, 492
.   : milestone, 471,
iast_FULL (551.376 µs) : 531, 572
.   : milestone, 551,
iast_GLOBAL (497.314 µs) : 476, 519
.   : milestone, 497,
iast_HARDCODED_SECRET_DISABLED (475.86 µs) : 455, 496
.   : milestone, 476,
iast_INACTIVE (449.955 µs) : 430, 470
.   : milestone, 450,
iast_TELEMETRY_OFF (473.27 µs) : 453, 494
.   : milestone, 473,
tracing (449.728 µs) : 429, 471
.   : milestone, 450,
section candidate
no_agent (370.251 µs) : 348, 392
.   : milestone, 370,
iast (472.493 µs) : 452, 493
.   : milestone, 472,
iast_FULL (533.609 µs) : 513, 554
.   : milestone, 534,
iast_GLOBAL (491.555 µs) : 471, 512
.   : milestone, 492,
iast_HARDCODED_SECRET_DISABLED (478.288 µs) : 458, 499
.   : milestone, 478,
iast_INACTIVE (449.61 µs) : 428, 471
.   : milestone, 450,
iast_TELEMETRY_OFF (470.93 µs) : 450, 492
.   : milestone, 471,
tracing (445.498 µs) : 425, 466
.   : milestone, 445,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.57 µs [347.015 µs, 388.124 µs] -
iast 471.259 µs [450.328 µs, 492.19 µs] 103.69 µs (28.2%)
iast_FULL 551.376 µs [530.795 µs, 571.958 µs] 183.807 µs (50.0%)
iast_GLOBAL 497.314 µs [475.897 µs, 518.732 µs] 129.745 µs (35.3%)
iast_HARDCODED_SECRET_DISABLED 475.86 µs [455.399 µs, 496.32 µs] 108.29 µs (29.5%)
iast_INACTIVE 449.955 µs [429.686 µs, 470.223 µs] 82.385 µs (22.4%)
iast_TELEMETRY_OFF 473.27 µs [452.827 µs, 493.713 µs] 105.701 µs (28.8%)
tracing 449.728 µs [428.659 µs, 470.796 µs] 82.158 µs (22.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 370.251 µs [348.058 µs, 392.443 µs] -
iast 472.493 µs [452.243 µs, 492.744 µs] 102.242 µs (27.6%)
iast_FULL 533.609 µs [513.255 µs, 553.963 µs] 163.358 µs (44.1%)
iast_GLOBAL 491.555 µs [471.096 µs, 512.013 µs] 121.304 µs (32.8%)
iast_HARDCODED_SECRET_DISABLED 478.288 µs [458.036 µs, 498.541 µs] 108.037 µs (29.2%)
iast_INACTIVE 449.61 µs [428.107 µs, 471.114 µs] 79.359 µs (21.4%)
iast_TELEMETRY_OFF 470.93 µs [450.26 µs, 491.6 µs] 100.679 µs (27.2%)
tracing 445.498 µs [425.258 µs, 465.737 µs] 75.247 µs (20.3%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~bcc535c3b1, baseline=1.32.0-SNAPSHOT~b708147d74
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.346 ms) : 1327, 1365
.   : milestone, 1346,
appsec (1.77 ms) : 1746, 1794
.   : milestone, 1770,
iast (1.543 ms) : 1520, 1566
.   : milestone, 1543,
profiling (1.541 ms) : 1516, 1566
.   : milestone, 1541,
tracing (1.504 ms) : 1480, 1527
.   : milestone, 1504,
section candidate
no_agent (1.35 ms) : 1331, 1369
.   : milestone, 1350,
appsec (1.77 ms) : 1746, 1793
.   : milestone, 1770,
iast (1.545 ms) : 1521, 1568
.   : milestone, 1545,
profiling (1.527 ms) : 1504, 1551
.   : milestone, 1527,
tracing (1.498 ms) : 1475, 1522
.   : milestone, 1498,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.346 ms [1.327 ms, 1.365 ms] -
appsec 1.77 ms [1.746 ms, 1.794 ms] 424.208 µs (31.5%)
iast 1.543 ms [1.52 ms, 1.566 ms] 197.043 µs (14.6%)
profiling 1.541 ms [1.516 ms, 1.566 ms] 195.022 µs (14.5%)
tracing 1.504 ms [1.48 ms, 1.527 ms] 157.616 µs (11.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.35 ms [1.331 ms, 1.369 ms] -
appsec 1.77 ms [1.746 ms, 1.793 ms] 419.883 µs (31.1%)
iast 1.545 ms [1.521 ms, 1.568 ms] 194.656 µs (14.4%)
profiling 1.527 ms [1.504 ms, 1.551 ms] 177.49 µs (13.1%)
tracing 1.498 ms [1.475 ms, 1.522 ms] 148.356 µs (11.0%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/selenium-support branch from 43991fc to 2c1886b Compare March 15, 2024 14:59
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/selenium-support branch from 2c1886b to 4badb54 Compare March 16, 2024 11:17
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review March 16, 2024 17:41
@nikita-tkachenko-datadog nikita-tkachenko-datadog added the tag: do not merge Do not merge changes label Mar 22, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog removed the tag: do not merge Do not merge changes label Apr 3, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit 1bb1766 into master Apr 4, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/selenium-support branch April 4, 2024 10:40
@github-actions github-actions bot added this to the 1.33.0 milestone Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tonyredondo tonyredondo tonyredondo approved these changes

@ygree ygree Awaiting requested review from ygree ygree is a code owner automatically assigned from DataDog/apm-java

@am312 am312 Awaiting requested review from am312 am312 is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility run-tests: all Run all tests

Projects

None yet

Milestone

1.33.0

Development

Successfully merging this pull request may close these issues.

3 participants

@nikita-tkachenko-datadog @tonyredondo