Skip to content

Add latest dependency tests and muzzle verifications for test frameworks based on JUnit 5 #5897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 28, 2023
Merged

Add latest dependency tests and muzzle verifications for test frameworks based on JUnit 5 #5897

merged 1 commit into from
Sep 28, 2023

Conversation

nikita-tkachenko-datadog
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Sep 15, 2023 •
edited
Loading

What Does This Do

Moves JUnit5-Cucumber and JUnit5-Spock instrumentation tests into their own dedicated modules, and adds muzzle verifications for those instrumentations.
Adds latest dependency tests for JUnit 5, JUnit5-Cucumber and JUnit5-Spock instrumentations.

Motivation

Improving reliability of these instrumentations.

@nikita-tkachenko-datadog nikita-tkachenko-datadog added tag: no release notes Changes to exclude from release notes comp: ci visibility Continuous Integration Visibility labels Sep 15, 2023
@pr-commenter
Copy link

pr-commenter bot commented Sep 15, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.22.0-SNAPSHOT~e043b194a5 1.22.0-SNAPSHOT~0da137fed4
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant appsec appsec

Summary

Found 2 performance improvements and 0 performance regressions! Performance is the same for 60 cases.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:petclinic:appsec:Remote Config better
[-30.579µs; -15.309µs] or [-4.648%; -2.327%]		 634.982µs 657.926µs
scenario:petclinic:iast:Remote Config better
[-69.722µs; -17.870µs] or [-11.729%; -3.006%]		 550.663µs 594.459µs
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.22.0-SNAPSHOT~0da137fed4, baseline=1.22.0-SNAPSHOT~e043b194a5

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.018 s) : 0, 1017866
Total [baseline] (9.341 s) : 0, 9341040
Agent [candidate] (1.017 s) : 0, 1016846
Total [candidate] (9.21 s) : 0, 9209787
section appsec
Agent [baseline] (1.094 s) : 0, 1094046
Total [baseline] (9.298 s) : 0, 9297876
Agent [candidate] (1.095 s) : 0, 1094724
Total [candidate] (9.305 s) : 0, 9305114
section iast
Agent [baseline] (1.143 s) : 0, 1143257
Total [baseline] (9.545 s) : 0, 9544871
Agent [candidate] (1.117 s) : 0, 1116887
Total [candidate] (9.345 s) : 0, 9345431
section profiling
Agent [baseline] (1.182 s) : 0, 1182111
Total [baseline] (9.479 s) : 0, 9479169
Agent [candidate] (1.188 s) : 0, 1187507
Total [candidate] (9.475 s) : 0, 9474566
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.018 s -
Agent appsec 1.094 s 76.18 ms (7.5%)
Agent iast 1.143 s 125.391 ms (12.3%)
Agent profiling 1.182 s 164.245 ms (16.1%)
Total tracing 9.341 s -
Total appsec 9.298 s -43.164 ms (-0.5%)
Total iast 9.545 s 203.831 ms (2.2%)
Total profiling 9.479 s 138.129 ms (1.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.017 s -
Agent appsec 1.095 s 77.878 ms (7.7%)
Agent iast 1.117 s 100.04 ms (9.8%)
Agent profiling 1.188 s 170.661 ms (16.8%)
Total tracing 9.21 s -
Total appsec 9.305 s 95.327 ms (1.0%)
Total iast 9.345 s 135.645 ms (1.5%)
Total profiling 9.475 s 264.779 ms (2.9%) 
gantt
    title petclinic - break down per module: candidate=1.22.0-SNAPSHOT~0da137fed4, baseline=1.22.0-SNAPSHOT~e043b194a5

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (630.177 ms) : 0, 630177
BytebuddyAgent [candidate] (630.863 ms) : 0, 630863
GlobalTracer [baseline] (297.339 ms) : 0, 297339
GlobalTracer [candidate] (295.976 ms) : 0, 295976
AppSec [baseline] (49.008 ms) : 0, 49008
AppSec [candidate] (48.585 ms) : 0, 48585
Remote Config [baseline] (682.176 µs) : 0, 682
Remote Config [candidate] (665.0 µs) : 0, 665
Telemetry [baseline] (6.146 ms) : 0, 6146
Telemetry [candidate] (6.012 ms) : 0, 6012
section appsec
BytebuddyAgent [baseline] (624.935 ms) : 0, 624935
BytebuddyAgent [candidate] (624.661 ms) : 0, 624661
GlobalTracer [baseline] (293.044 ms) : 0, 293044
GlobalTracer [candidate] (293.857 ms) : 0, 293857
AppSec [baseline] (135.22 ms) : 0, 135220
AppSec [candidate] (135.431 ms) : 0, 135431
Remote Config [baseline] (657.926 µs) : 0, 658
Remote Config [candidate] (634.982 µs) : 0, 635
Telemetry [baseline] (5.869 ms) : 0, 5869
Telemetry [candidate] (5.866 ms) : 0, 5866
section iast
BytebuddyAgent [baseline] (756.583 ms) : 0, 756583
BytebuddyAgent [candidate] (738.61 ms) : 0, 738610
GlobalTracer [baseline] (283.066 ms) : 0, 283066
GlobalTracer [candidate] (277.416 ms) : 0, 277416
AppSec [baseline] (46.756 ms) : 0, 46756
AppSec [candidate] (45.74 ms) : 0, 45740
Remote Config [baseline] (594.459 µs) : 0, 594
Remote Config [candidate] (550.663 µs) : 0, 551
Telemetry [baseline] (5.988 ms) : 0, 5988
Telemetry [candidate] (5.766 ms) : 0, 5766
IAST [baseline] (14.922 ms) : 0, 14922
IAST [candidate] (14.34 ms) : 0, 14340
section profiling
BytebuddyAgent [baseline] (637.218 ms) : 0, 637218
BytebuddyAgent [candidate] (641.023 ms) : 0, 641023
GlobalTracer [baseline] (355.142 ms) : 0, 355142
GlobalTracer [candidate] (356.213 ms) : 0, 356213
AppSec [baseline] (48.997 ms) : 0, 48997
AppSec [candidate] (48.945 ms) : 0, 48945
Remote Config [baseline] (662.451 µs) : 0, 662
Remote Config [candidate] (662.422 µs) : 0, 662
Telemetry [baseline] (6.12 ms) : 0, 6120
Telemetry [candidate] (6.063 ms) : 0, 6063
ProfilingAgent [baseline] (80.991 ms) : 0, 80991
ProfilingAgent [candidate] (81.32 ms) : 0, 81320
Profiling [baseline] (81.015 ms) : 0, 81015
Profiling [candidate] (81.344 ms) : 0, 81344
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.22.0-SNAPSHOT~0da137fed4, baseline=1.22.0-SNAPSHOT~e043b194a5

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.004 s) : 0, 1004196
Total [baseline] (8.656 s) : 0, 8655688
Agent [candidate] (1.01 s) : 0, 1009873
Total [candidate] (8.681 s) : 0, 8680764
section appsec
Agent [baseline] (1.101 s) : 0, 1100523
Total [baseline] (8.775 s) : 0, 8775178
Agent [candidate] (1.096 s) : 0, 1095724
Total [candidate] (8.767 s) : 0, 8766869
section iast
Agent [baseline] (1.118 s) : 0, 1117888
Total [baseline] (9.199 s) : 0, 9198773
Agent [candidate] (1.116 s) : 0, 1116068
Total [candidate] (9.267 s) : 0, 9266752
section profiling
Agent [baseline] (1.179 s) : 0, 1179097
Total [baseline] (8.884 s) : 0, 8883739
Agent [candidate] (1.186 s) : 0, 1185646
Total [candidate] (8.939 s) : 0, 8938843
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.004 s -
Agent appsec 1.101 s 96.327 ms (9.6%)
Agent iast 1.118 s 113.691 ms (11.3%)
Agent profiling 1.179 s 174.901 ms (17.4%)
Total tracing 8.656 s -
Total appsec 8.775 s 119.489 ms (1.4%)
Total iast 9.199 s 543.085 ms (6.3%)
Total profiling 8.884 s 228.051 ms (2.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.01 s -
Agent appsec 1.096 s 85.851 ms (8.5%)
Agent iast 1.116 s 106.195 ms (10.5%)
Agent profiling 1.186 s 175.773 ms (17.4%)
Total tracing 8.681 s -
Total appsec 8.767 s 86.105 ms (1.0%)
Total iast 9.267 s 585.988 ms (6.8%)
Total profiling 8.939 s 258.079 ms (3.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.22.0-SNAPSHOT~0da137fed4, baseline=1.22.0-SNAPSHOT~e043b194a5

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (622.524 ms) : 0, 622524
BytebuddyAgent [candidate] (624.008 ms) : 0, 624008
GlobalTracer [baseline] (292.104 ms) : 0, 292104
GlobalTracer [candidate] (295.801 ms) : 0, 295801
AppSec [baseline] (48.585 ms) : 0, 48585
AppSec [candidate] (49.148 ms) : 0, 49148
Remote Config [baseline] (668.608 µs) : 0, 669
Remote Config [candidate] (675.663 µs) : 0, 676
Telemetry [baseline] (6.006 ms) : 0, 6006
Telemetry [candidate] (6.013 ms) : 0, 6013
section appsec
BytebuddyAgent [baseline] (629.029 ms) : 0, 629029
BytebuddyAgent [candidate] (624.243 ms) : 0, 624243
GlobalTracer [baseline] (295.138 ms) : 0, 295138
GlobalTracer [candidate] (295.315 ms) : 0, 295315
AppSec [baseline] (135.38 ms) : 0, 135380
AppSec [candidate] (135.506 ms) : 0, 135506
Remote Config [baseline] (657.731 µs) : 0, 658
Remote Config [candidate] (643.574 µs) : 0, 644
Telemetry [baseline] (5.897 ms) : 0, 5897
Telemetry [candidate] (5.899 ms) : 0, 5899
section iast
BytebuddyAgent [baseline] (739.867 ms) : 0, 739867
BytebuddyAgent [candidate] (737.77 ms) : 0, 737770
GlobalTracer [baseline] (277.031 ms) : 0, 277031
GlobalTracer [candidate] (277.403 ms) : 0, 277403
AppSec [baseline] (45.776 ms) : 0, 45776
AppSec [candidate] (45.836 ms) : 0, 45836
Remote Config [baseline] (552.097 µs) : 0, 552
Remote Config [candidate] (544.454 µs) : 0, 544
Telemetry [baseline] (5.74 ms) : 0, 5740
Telemetry [candidate] (5.708 ms) : 0, 5708
IAST [baseline] (14.325 ms) : 0, 14325
IAST [candidate] (14.279 ms) : 0, 14279
section profiling
BytebuddyAgent [baseline] (637.023 ms) : 0, 637023
BytebuddyAgent [candidate] (640.278 ms) : 0, 640278
GlobalTracer [baseline] (354.257 ms) : 0, 354257
GlobalTracer [candidate] (356.207 ms) : 0, 356207
AppSec [baseline] (48.417 ms) : 0, 48417
AppSec [candidate] (48.375 ms) : 0, 48375
Remote Config [baseline] (651.899 µs) : 0, 652
Remote Config [candidate] (653.442 µs) : 0, 653
Telemetry [baseline] (6.061 ms) : 0, 6061
Telemetry [candidate] (6.089 ms) : 0, 6089
ProfilingAgent [baseline] (79.691 ms) : 0, 79691
ProfilingAgent [candidate] (80.725 ms) : 0, 80725
Profiling [baseline] (79.716 ms) : 0, 79716
Profiling [candidate] (80.749 ms) : 0, 80749
Loading

Load

Parameters

Baseline Candidate
commit 1.22.0-SNAPSHOT~e043b194a5 1.22.0-SNAPSHOT~0da137fed4
config baseline candidate
end_time 2023-09-27T15:29:24 2023-09-27T15:47:32
start_time 2023-09-27T15:29:06 2023-09-27T15:47:14
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.22.0-SNAPSHOT~0da137fed4, baseline=1.22.0-SNAPSHOT~e043b194a5
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.33 ms) : 1311, 1349
.   : milestone, 1330,
appsec (1.687 ms) : 1662, 1711
.   : milestone, 1687,
iast (1.476 ms) : 1451, 1500
.   : milestone, 1476,
profiling (1.455 ms) : 1429, 1480
.   : milestone, 1455,
tracing (1.456 ms) : 1431, 1480
.   : milestone, 1456,
section candidate
no_agent (1.331 ms) : 1312, 1350
.   : milestone, 1331,
appsec (1.685 ms) : 1660, 1709
.   : milestone, 1685,
iast (1.471 ms) : 1446, 1496
.   : milestone, 1471,
profiling (1.499 ms) : 1474, 1524
.   : milestone, 1499,
tracing (1.428 ms) : 1404, 1453
.   : milestone, 1428,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.33 ms [1.311 ms, 1.349 ms] -
appsec 1.687 ms [1.662 ms, 1.711 ms] 356.434 µs (26.8%)
iast 1.476 ms [1.451 ms, 1.5 ms] 145.758 µs (11.0%)
profiling 1.455 ms [1.429 ms, 1.48 ms] 124.618 µs (9.4%)
tracing 1.456 ms [1.431 ms, 1.48 ms] 125.55 µs (9.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.331 ms [1.312 ms, 1.35 ms] -
appsec 1.685 ms [1.66 ms, 1.709 ms] 353.097 µs (26.5%)
iast 1.471 ms [1.446 ms, 1.496 ms] 139.254 µs (10.5%)
profiling 1.499 ms [1.474 ms, 1.524 ms] 167.884 µs (12.6%)
tracing 1.428 ms [1.404 ms, 1.453 ms] 96.94 µs (7.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.22.0-SNAPSHOT~0da137fed4, baseline=1.22.0-SNAPSHOT~e043b194a5
    dateFormat X
    axisFormat %s
section baseline
no_agent (364.006 µs) : 344, 384
.   : milestone, 364,
appsec (675.591 µs) : 654, 697
.   : milestone, 676,
iast (469.536 µs) : 449, 490
.   : milestone, 470,
iast_FULL (522.045 µs) : 501, 543
.   : milestone, 522,
iast_INACTIVE (437.636 µs) : 417, 459
.   : milestone, 438,
profiling (439.044 µs) : 418, 460
.   : milestone, 439,
tracing (426.94 µs) : 407, 447
.   : milestone, 427,
section candidate
no_agent (354.543 µs) : 335, 374
.   : milestone, 355,
appsec (670.198 µs) : 650, 691
.   : milestone, 670,
iast (457.941 µs) : 437, 479
.   : milestone, 458,
iast_FULL (523.116 µs) : 503, 544
.   : milestone, 523,
iast_INACTIVE (433.317 µs) : 413, 454
.   : milestone, 433,
profiling (438.53 µs) : 418, 459
.   : milestone, 439,
tracing (431.477 µs) : 411, 452
.   : milestone, 431,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.006 µs [344.359 µs, 383.653 µs] -
appsec 675.591 µs [654.073 µs, 697.11 µs] 311.586 µs (85.6%)
iast 469.536 µs [448.619 µs, 490.453 µs] 105.53 µs (29.0%)
iast_FULL 522.045 µs [500.648 µs, 543.443 µs] 158.039 µs (43.4%)
iast_INACTIVE 437.636 µs [416.767 µs, 458.506 µs] 73.63 µs (20.2%)
profiling 439.044 µs [418.111 µs, 459.977 µs] 75.038 µs (20.6%)
tracing 426.94 µs [406.551 µs, 447.329 µs] 62.934 µs (17.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 354.543 µs [335.007 µs, 374.078 µs] -
appsec 670.198 µs [649.581 µs, 690.816 µs] 315.656 µs (89.0%)
iast 457.941 µs [437.138 µs, 478.745 µs] 103.399 µs (29.2%)
iast_FULL 523.116 µs [502.643 µs, 543.589 µs] 168.574 µs (47.5%)
iast_INACTIVE 433.317 µs [412.533 µs, 454.101 µs] 78.775 µs (22.2%)
profiling 438.53 µs [417.845 µs, 459.214 µs] 83.987 µs (23.7%)
tracing 431.477 µs [410.697 µs, 452.258 µs] 76.935 µs (21.7%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/testng-itr-unskippable branch from e3e2570 to dab2aea Compare September 25, 2023 12:57
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/latest-dep-tests-junit5 branch from d63b8de to d28a9b2 Compare September 25, 2023 12:58
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/testng-itr-unskippable branch from dab2aea to bca3bba Compare September 26, 2023 13:07
Base automatically changed from nikita-tkachenko/testng-itr-unskippable to master September 27, 2023 15:04
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/latest-dep-tests-junit5 branch from d28a9b2 to 0da137f Compare September 27, 2023 15:05
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review September 27, 2023 16:02
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit 98b7911 into master Sep 28, 2023
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/latest-dep-tests-junit5 branch September 28, 2023 15:27
@github-actions github-actions bot added this to the 1.22.0 milestone Sep 28, 2023
jandro996 pushed a commit that referenced this pull request Oct 3, 2023
@nikita-tkachenko-datadog @jandro996
Add latest dependency tests and muzzle verifications for test framewo…
2f58a0e 
…rks based on JUnit 5 (#5897)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@juan-fernandez juan-fernandez juan-fernandez approved these changes

@tonyredondo tonyredondo Awaiting requested review from tonyredondo tonyredondo is a code owner automatically assigned from DataDog/ci-app-libraries-java

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

@ygree ygree Awaiting requested review from ygree ygree is a code owner automatically assigned from DataDog/apm-java

Assignees
No one assigned
Labels
comp: ci visibility Continuous Integration Visibility tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
1.22.0
Development

Successfully merging this pull request may close these issues.
2 participants
@nikita-tkachenko-datadog @juan-fernandez